23 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 38ae99fb8e feat(hosts): the hottest sensor is named with its chip, and the host-reading cutoffs are read and set over MCP (#6806)
CI / lint (push) Successful in 1s
CI / unit (push) Successful in 1m39s
CI / integration (push) Successful in 2m2s
CI / publish (push) Successful in 1m12s
Agent 1.27.0 sends each temperature sensor's hwmon chip. Steward keeps the
chip per host and label beside the reading, so a sensor's metric key stays
its label, and the strip and the heat alert name it as "NVMe · Sensor 2"
rather than a bare "Sensor 2".

get_thresholds (viewer) and set_thresholds (admin, audited) read and change
the warn/crit cutoffs CPU, memory, disk, load and temperature are graded by,
as Settings → Thresholds does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 17:39:31 -05:00
bvandeusenandClaude Opus 5.5 6eeb4039d7 feat(hosts): the live strip shows the hottest sensor's temperature, get_host marks sensors that don't count, and the Monitors card's button reads "Add" (#6803)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m58s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 1m7s
- The host page's live strip gains a Temperature tile: temp_c_max (real
  thermometers only), its last 6 h and the sensor setting it, graded by
  Settings → Thresholds and counted in the header's worst reading, in °C.
- MCP get_host: each temperature carries `counted`; an unwired input is
  `counted: false` with `not_counted_because`, listed after the real ones.
  host_agent registers the judge (judge_sensors), so core imports no plugin.
- The Monitors card's "Add monitor" wrapped into two lines beside the
  uptime; it reads "Add", titled "Add a monitor for this host".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 16:00:49 -05:00
bvandeusenandClaude Opus 5.5 c38fd68e48 feat(ui): a host's "Full metrics" heads its page, and a resource's anomalies are one line with a way to review them (#6797)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m38s
CI / integration (push) Successful in 2m0s
CI / publish (push) Successful in 1m10s
The host page's link to the full metrics sat in the Agent card, under the
anomaly card and the vitals strip, and was missed. It now heads the vitals
strip at the top of the page.

The anomaly card on the host, monitor, container and Traefik pages listed
every anomaly of the last 30 days with its review form, reviewed ones
included, and took over the top of the page. It is now one line: how many
wait for review, which are out of their normal now, and a Review button to
Alerts → Anomalies narrowed to that resource (?source=&resource=), where a
review returns. It is hidden when nothing waits and nothing is going on.
The callers fetch up to 100 rows so the count isn't capped at 10 or 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:09:18 -05:00
bvandeusenandClaude Opus 5.5 12a5f83c67 fix(host-agent): a large update records its package count (#6794)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m38s
CI / integration (push) Successful in 2m1s
CI / publish (push) Successful in 20s
The root helper counted updated packages from the last 2,000 characters
of the package manager's output. Every manager prints its count (apt's
"N upgraded, N newly installed", dnf's "Upgrade N Packages", zypper's
"N packages to upgrade", pacman's "Packages (N)") before the long log of
what it then does, so a large update pushed the line out of the tail
and the job recorded no count. Docker03's 2026-11-01 job applied 17
updates and reported none; Docker02's single package was counted.

The update steps now read their whole output (run(keep=None)), and only
the failure text and the job's detail are cut short. Agent 1.26.1, so
hosts show the helper as an update to take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 23:03:07 -05:00
bvandeusenandClaude Opus 5.5 c3072312a4 fix(anomalies): one slow reading no longer opens an anomaly (#6793)
CI / integration (push) Successful in 2m3s
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m38s
CI / publish (push) Successful in 1m30s
A single tall reading lifted a 15-minute average past the usual height
on its own, so one slow request opened a row that stayed open until the
reading rolled out of the window. Four of the seven rows that occurred
after the 10-30 detection change were this: HA web (one 6.7 s check),
error-pages, codex and mailcow-svc latency.

detect() now opens by height only when the window is still past the
usual height, and still departs, with its highest reading left out. A
window that fails that falls through to the held test, which now needs
at least HELD_FEWEST (3) readings behind its median, so a lone reading
on a quiet router opens nothing either way.

The median and the topless average share one per-key read (_per_key).
The architecture doc and get_metric_normal's description state the
rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:41:29 -05:00
bvandeusenandClaude Opus 5.5 19f405e55a test(homeassistant): the page's wording on who reboots after an OS update (#6755)
CI / unit (push) Successful in 1m36s
CI / lint (push) Successful in 1s
CI / integration (push) Successful in 2m0s
CI / publish (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 19:22:37 -05:00
bvandeusenandClaude Opus 5.5 ecc59cc882 feat(homeassistant): a scheduled OS update reboots the machine and is done once the new version runs (#6755)
CI / lint (push) Successful in 1s
CI / unit (push) Successful in 1m37s
CI / integration (push) Failing after 2m0s
CI / publish (push) Skipped
Home Assistant writes an OS update and leaves the reboot to someone, so a scheduled
one never finished unattended. The schedule now asks for the reboot once, follows it
through Home Assistant going away, and counts it done when the Supervisor reports a
different version with nothing pending. A machine that doesn't come back or falls
back to the old version fails after 20 minutes, which ends the slot and raises
update_failed. Install on the page and through MCP still leave the reboot to the person.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 19:15:03 -05:00
bvandeusenandClaude Opus 5.5 4c57735dd9 fix(homeassistant): storage reads used of total with its percentage, and an OS update is said to wait on a reboot (#6753, #6754)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m59s
CI / publish (push) Successful in 19s
The tile shows Home Assistant's own used figure beside the total, so it matches
Home Assistant's storage page. An OS update doesn't reboot the machine: Supervisor
writes the image, reports the new version and raises a repair asking for a reboot.
The confirmation, schedule warning, MCP docs and plugin docs now say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:51:28 -05:00
bvandeusenandClaude Opus 5.5 5ffececae5 feat(homeassistant): the page shows the device's storage and how full it is (#6753)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m59s
CI / publish (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 15:03:13 -05:00
bvandeusenandClaude Opus 5.5 ee9913b6d0 fix(prometheus): the migration removing sign-in targets runs on a new database, where the metric tables may not exist yet (#6747)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m38s
CI / integration (push) Successful in 1m57s
CI / publish (push) Successful in 1m15s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 14:24:00 -05:00
bvandeusenandClaude Opus 5.5 f7a856cb90 feat(homeassistant): it shows system problems and installs updates on a weekly slot, and leaves device readings alone (#6748, #6749, #6747)
CI / unit (push) Successful in 1m37s
CI / integration (push) Failing after 1m54s
CI / publish (push) Skipped
CI / lint (push) Successful in 2s
The connection's job is keeping Home Assistant running. Each poll now reads open
repairs, Supervisor health, disk, recovery mode and add-ons that should be running
over its websocket, for the page, the overview and the problems_open, disk_free_pct
and update_failed metrics. That needs a websocket client, so websockets joins the
dependencies.

A weekly slot, set on the page or through MCP and off until then, installs add-ons,
then Core, then the OS, each after its own confirmed backup, ending at the first
failure. The slot arithmetic moves from the host agent's cadence to steward.core.weekly
so both share it.

The device scrape goes: the page's Metrics section and a target's token_from from
4c9ed02 are removed, and a migration deletes targets using it, as they'd otherwise ask
with no token. A refused token still stops a scrape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 14:16:42 -05:00
bvandeusenandClaude Opus 5.5 4c9ed02652 feat(homeassistant): its page offers the metrics scrape under the same sign-in, so one user covers everything (#6744)
CI / lint (push) Successful in 1s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m58s
CI / publish (push) Successful in 1m13s
A Prometheus target's token_from names the plugin that supplies its token at each
scrape. Read its metrics makes the target; Use this sign-in moves one at that address
across, keeping its name, filters and history. A refusal stops the connection, so one
Try again or sign-in starts everything, and the sign-in is only sent to its own address.
A 404 says how to turn on Home Assistant's Prometheus integration, as Steward doesn't
change Home Assistant. Refused also joins the volatile columns a backup check ignores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:06:55 -05:00
bvandeusenandClaude Opus 5.5 66a5dd12fd fix(prometheus): a refused token stops the scraping, as Home Assistant bans an address after a few refusals (#6355)
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m55s
CI / publish (push) Successful in 20s
CI / lint (push) Successful in 2s
Also names the Home Assistant connection by its address in the audit log and
timeline, where its id showed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 09:41:53 -05:00
bvandeusenandClaude Opus 5.5 a1dfa7b821 test: an invite's zone may be EDT or EST, and an unsent install counts from a recent backup (#6355)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m53s
CI / unit (push) Successful in 1m37s
CI / publish (push) Successful in 1m13s
The invite test asserted EDT for a time a day from now, so it failed from today:
a one-day link now ends after New York's clocks change on 1 November. It would
have stayed red until March.

The Home Assistant test used a fixed backup time, which put the unsent install
past its deadline at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:19:12 -04:00
bvandeusenandClaude Opus 5.5 0aed863c67 feat(homeassistant): an admin signs in to Home Assistant, with nothing added to it (#6355)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m37s
CI / integration (push) Failing after 1m55s
CI / publish (push) Skipped
Replaces the read-only token and the webhook automation, which every install
would have had to paste into its own Home Assistant. Connect now goes to Home
Assistant's own sign-in page, and Steward keeps the refresh token that comes
back, encrypted, and calls backup.create_automatic and update.install directly.

The connection page warns that the stored credential has the same access as
the user who signed in, and recommends a service account so Steward's actions
are easy to tell apart in Home Assistant's logs. A user who isn't an
administrator gets the status page and no Install.

Those actions answer only when finished, so a request cut off once sent counts
as sent and the outcome is read from the entities. Removing the connection, or
signing in again, ends the old sign-in on Home Assistant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:14:34 -04:00
bvandeusenandClaude Opus 5.5 9c9e14e79c fix(homeassistant): its MCP tools register, stating read_only as the server requires (#6355)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m39s
CI / integration (push) Successful in 1m51s
CI / publish (push) Successful in 19s
Without it the plugin's hook raised at boot and left it with no tools. The
test's stand-in for register gave read_only a default, so now both arguments
are required there too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:25:51 -04:00
bvandeusenandClaude Opus 5.5 fe9bba1042 test(homeassistant): its page is a section's top page, and a test's unused import goes (#6355)
CI / unit (push) Successful in 1m37s
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m53s
CI / publish (push) Successful in 1m20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:16:46 -04:00
bvandeusenandClaude Opus 5.5 b2a6e92ed3 feat(homeassistant): waiting updates, backup age, and updates installed after a confirmed backup (#6355)
CI / unit (push) Failing after 1m42s
CI / integration (push) Successful in 1m51s
CI / publish (push) Skipped
CI / lint (push) Failing after 2s
A new plugin reads Home Assistant with a read-only token every 5 minutes and
shows which Core, OS and add-on updates are waiting and how old the last backup
is. Install takes a backup through a local-only webhook, confirms Home
Assistant's last-backup time moved, installs, and follows the update to its end,
through Home Assistant's and Steward's own restarts. One runs at a time. Each
ends as one timeline entry.

Steward holds nothing that can act on the house: the webhook's automation can
only take a backup or install an allowed update (System 231's ruling). A refused
token stops polling, as Home Assistant bans an address after a few refusals.

Milestone 576 steps 1 to 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:09:27 -04:00
bvandeusenandClaude Opus 5.5 8b9b7fc11d chore(docker): the history move logs which services its old keys belong to (#6718)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m35s
CI / integration (push) Successful in 1m47s
CI / publish (push) Successful in 19s
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". The move found 391,091
hourly-only task names against 51 services and about 400 keys with raw samples.
Log the services with the most keys and a few names, to read before those keys
start moving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:13:20 -04:00
bvandeusenandClaude Opus 5.5 acc487f9c3 fix(docker): the move of task-named history rests between passes (#6718)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m34s
CI / integration (push) Successful in 1m49s
CI / publish (push) Successful in 20s
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". Live, the move took
VDNT-Docker02's I/O pressure (psi_io_full_avg60) from about 1% to about 23%,
and passes of 5000 raw samples outran their 15 s slot. Each moved sample
rewrites its row and every index on it.

Batches are 1000 samples, and after a pass the task waits four times as long
as it took, so it holds the disk for about a fifth of the time and the move
slows if the disk does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:01:10 -04:00
bvandeusenandClaude Opus 5.5 4689386b69 fix(docker): the move of task-named history finishes many small keys in one pass (#6718)
CI / integration (push) Successful in 1m48s
CI / publish (push) Successful in 20s
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m34s
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". Live, the move found 391,440
keys, about 2,500 of them with raw samples, and took one key per 15 s pass:
about 68 days, in sorted order, so recent history could wait months while
charts and normals read thin.

A pass now runs for up to 2 s, and keys with raw samples go first. Old hourly
keys take a few statements each. A key with no anomalies no longer rewrites the
service's timeline entries. The start logs how many keys each table holds, as
that count is far higher than expected and not explained yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:53:43 -04:00
bvandeusenandClaude Opus 5.5 177ea12c9b fix(anomalies): a burst within what a metric reaches in most hours opens nothing (#6727)
CI / unit (push) Successful in 1m36s
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m50s
CI / publish (push) Successful in 1m11s
#6727 "A metric that bursts now and then opens anomalies at hours whose usual
peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that
bursts in under about 1 hour in 10 reads 0 there however often it bursts across
the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual
peak, so most rows aren't outside the norm" therefore still opened on those.

usual_anywhere is the median, across the hours of the day, of each hour's
highest peak. Height is now judged against the greater of it and the slot's
usual peak. Bursts at any hour lift the median; a job at a time of its own
raises a few hours and leaves it low, so the same level at another time of day
is still an anomaly, as System 182 rules. A held level still opens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:45:16 -04:00
bvandeusenandClaude Opus 5.5 db39fc2604 feat(docker): a swarm service's metrics stay under one key across its tasks (#6718)
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". The operator: "the metrics
should be keys to the service name not the task name... as that's what's
consistent between updates."

CPU, memory, restart count and health now go under "swarm/<service>", so a
normal, its anomalies and an alert rule carry across a redeploy or a move
between nodes. Network rates stay "<host>/<service>", the task id gone, as they
answer who on a host is sending. Tasks of a multi-task service share the key,
so its level reads as an average across them (of 51 live services, only
authentik-worker and portainer_agent run more than one task).

History under task names moves in a background task, 5000 raw samples every
15 s, not a migration: the raw table holds about 2M such rows and the app
would wait on it at boot. Hourly rows combine where two tasks share an hour,
and rows still to review merge to one per metric.

A service's anomalies no longer reach a host's page, as it has no one host;
get_anomaly reads what happened around one from the service's own events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:45:16 -04:00
93 changed files with 4468 additions and 210 deletions
+1
View File
@@ -126,6 +126,7 @@ See `docs/plugins/` for a full plugin development guide.
| `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide | | `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide |
| `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples | | `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples |
| `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe | | `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe |
| `docs/plugins/homeassistant.md` | Home Assistant plugin: waiting updates, backup age, and installing updates after a confirmed backup |
| `docs/plugins/syslog.md` | Syslog plugin: sources, patterns and alerts, publishing port 5514 (ingress or host mode under Swarm) | | `docs/plugins/syslog.md` | Syslog plugin: sources, patterns and alerts, publishing port 5514 (ingress or host mode under Swarm) |
| `docs/plugins/prometheus-widgets.md` | Prometheus widget templates: writing, sharing and copying one in, every block, the format reference | | `docs/plugins/prometheus-widgets.md` | Prometheus widget templates: writing, sharing and copying one in, every block, the format reference |
| `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase | | `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase |
+3 -3
View File
@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`). A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also past the usual peak for that hour, or held: the median of the last hour is over the same line. A burst of a few minutes within the usual peak lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`): Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing. Height is judged with the window's highest reading left out, so one slow request can't carry a window past it, and a held level needs at least three readings behind its median, so a lone reading never opens an anomaly (`steward/core/anomalies.py`):
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`). - `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`. - A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review. - Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
@@ -118,14 +118,14 @@ What learns a normal, and the floor each departure must clear (Settings → Thre
| Host CPU | `host_agent` / `cpu_pct` | 10 points | | Host CPU | `host_agent` / `cpu_pct` | 10 points |
| Host memory | `host_agent` / `mem_used_pct` | 5 points | | Host memory | `host_agent` / `mem_used_pct` | 5 points |
| Host load per core | `host_agent` / `load_pct` (load_1m ÷ cores × 100, recorded at ingest) | 10 points | | Host load per core | `host_agent` / `load_pct` (load_1m ÷ cores × 100, recorded at ingest) | 10 points |
| Container CPU | `docker` / `cpu_pct` (100% is one core) | 10 points of one core | | Container CPU | `docker` / `cpu_pct` (100% is one core; a swarm service's under `swarm/<service>`, shared by its tasks) | 10 points of one core |
| Container memory | `docker` / `mem_used_pct` (its processes' own memory, file cache left out, of its limit or of host RAM when it has none) | 5 points | | Container memory | `docker` / `mem_used_pct` (its processes' own memory, file cache left out, of its limit or of host RAM when it has none) | 5 points |
| Monitor response time | `<type>` / `response_ms` | 100 ms | | Monitor response time | `<type>` / `response_ms` | 100 ms |
| Router 4xx share, p95 | `traefik` / `error_rate_4xx_pct`, `latency_p95_ms` | 2 points, 50 ms | | Router 4xx share, p95 | `traefik` / `error_rate_4xx_pct`, `latency_p95_ms` | 2 points, 50 ms |
Host metrics learn with `parts=False`: the host learns, its cores (`<host>:core<N>`) don't. Disk space, a monitor that is down, an agent gone quiet and a router's 5xx share are failures and keep their fixed cutoffs (a router's shares are judged only over enough requests). Host metrics learn with `parts=False`: the host learns, its cores (`<host>:core<N>`) don't. Disk space, a monitor that is down, an agent gone quiet and a router's 5xx share are failures and keep their fixed cutoffs (a router's shares are judged only over enough requests).
Pages grade by the normal through `anomalies.graded()` (hosts' `host_grades` and `worst_reading(normals=)`, `docker.service.usage(thresholds=, norms=)`, `monitors.service.latency_grade`): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages list their anomalies with `_anomaly_card.html`. Pages grade by the normal through `anomalies.graded()` (hosts' `host_grades` and `worst_reading(normals=)`, `docker.service.usage(thresholds=, norms=)`, `monitors.service.latency_grade`): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages each carry a one-line `_anomaly_card.html`. It says how many anomalies wait for review and names any that are out of their normal now. Its Review button opens Alerts → Anomalies narrowed to that resource (`?source=&resource=`), where they are reviewed.
--- ---
+3 -1
View File
@@ -76,6 +76,7 @@ Each role includes the tools of the roles above it in this list.
| Monitors | `list_monitors`, `get_monitor` | | Monitors | `list_monitors`, `get_monitor` |
| Alerts | `list_alerts`, `get_alert_history`, `list_alert_rules`, `get_alert_rule_options`, `list_maintenance_windows` | | Alerts | `list_alerts`, `get_alert_history`, `list_alert_rules`, `get_alert_rule_options`, `list_maintenance_windows` |
| Metric anomalies | `list_anomalies` (metrics that left their normal for the hour, to review first), `get_anomaly` (one anomaly with each occurrence against what is usual at that hour, the metric around it, what else happened on its host — alerts, Ansible runs, container deaths and starts, maintenance, reboots, other anomalies — and how earlier ones on the same metric were judged), `get_metric_normal` (the learned normal slot by slot, and the level that would be flagged), `get_anomaly_settings` (the review window and each floor) | | Metric anomalies | `list_anomalies` (metrics that left their normal for the hour, to review first), `get_anomaly` (one anomaly with each occurrence against what is usual at that hour, the metric around it, what else happened on its host — alerts, Ansible runs, container deaths and starts, maintenance, reboots, other anomalies — and how earlier ones on the same metric were judged), `get_metric_normal` (the learned normal slot by slot, and the level that would be flagged), `get_anomaly_settings` (the review window and each floor) |
| Thresholds | `get_thresholds` (the warn and crit cutoffs CPU, memory, disk, load and temperature are graded by on the pages) |
| Automation | `list_playbooks`, `get_playbook`, `list_inventory`, `list_ansible_runs`, `get_ansible_run`, `wait_for_run`, `list_schedules` | | Automation | `list_playbooks`, `get_playbook`, `list_inventory`, `list_ansible_runs`, `get_ansible_run`, `wait_for_run`, `list_schedules` |
| Audit | `list_audit_events` | | Audit | `list_audit_events` |
| Timeline | `get_timeline` (everything that happened, newest first, in one list: alerts firing and clearing, playbook runs and what they changed, maintenance, reboots, deploys and crash-loops, reviewed anomalies, changes recorded by hand, changes to Steward's settings; by host, area or kind, and kept after the records it came from age out) | | Timeline | `get_timeline` (everything that happened, newest first, in one list: alerts firing and clearing, playbook runs and what they changed, maintenance, reboots, deploys and crash-loops, reviewed anomalies, changes recorded by hand, changes to Steward's settings; by host, area or kind, and kept after the records it came from age out) |
@@ -102,7 +103,8 @@ Each role includes the tools of the roles above it in this list.
### Admin: destructive and remediation ### Admin: destructive and remediation
`set_anomaly_settings` (the anomaly review window and each normal's floor, as `set_anomaly_settings` (the anomaly review window and each normal's floor, as
Settings → Thresholds sets them), `set_alert_action` (the playbook a rule runs when it fires), `delete_monitor`, Settings → Thresholds sets them), `set_thresholds` (the warn and crit cutoffs host
readings are graded by, as the same page sets them), `set_alert_action` (the playbook a rule runs when it fires), `delete_monitor`,
`delete_alert_rule`, `prune_docker`, `run_maintenance` (update a host now, rebooting `delete_alert_rule`, `prune_docker`, `run_maintenance` (update a host now, rebooting
only if the update needs it, as the host page's Update now does). only if the update needs it, as the host page's Update now does).
+54
View File
@@ -0,0 +1,54 @@
# Home Assistant plugin
Keeps a Home Assistant instance running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates, on a click or a weekly schedule, each only after a fresh backup is confirmed. It looks after Home Assistant itself and doesn't read your devices. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API.
## Setting it up
1. In Home Assistant, turn on automatic backups (Settings → System → Backups).
2. Make a user for Steward (Settings → People → Add person), as an administrator if Steward should install updates.
3. On Steward's Home Assistant page, choose Connect, enter Home Assistant's address, and continue to its sign-in page. Sign in as that user.
Your browser and Steward both need to reach Home Assistant at that address. Home Assistant sends your browser back to Steward's Public base URL (Settings → General), so open Steward at that address.
## What Steward holds
**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates. It is stored encrypted.
We recommend a service account, a user made just for Steward, rather than your own. What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.
Installing needs an administrator, because Home Assistant only lets administrators take backups. With any other user, Steward shows status only. Steward learns which it has by asking once after sign-in, and a user who isn't an administrator is refused that one request, which Home Assistant may show as a failed login notification.
Removing the connection ends the sign-in on Home Assistant. You can also end it yourself under Security on that user's profile, and Steward then stops asking until you sign in again. Home Assistant bans an address after a few refused requests, so Steward stops after one refusal until you sign in again or choose Try again.
## Installing an update
Install asks once, then Steward:
1. Asks for a backup and waits until Home Assistant's last successful backup time moves. If it doesn't, nothing is installed.
2. Asks for the update and follows it until the new version is installed, or until a deadline: 20 minutes for an add-on, 30 for Core, 45 for the OS.
One runs at a time, and it carries on if Steward restarts. Firmware and custom integrations are left to Home Assistant, and Supervisor updates itself.
An OS update takes effect only after the machine reboots, and Install leaves that to you. Steward records the update as done once Home Assistant reports the new version, which it does before the reboot. Home Assistant then raises a repair asking for the reboot, which appears under System problems until you do it.
## System problems
Each read also asks Home Assistant whether the system itself has a problem, and the page and the overview show what it says:
- open repairs, from Settings → System → Repairs
- the Supervisor calling the system unhealthy or unsupported
- less than 10% of the disk free
- starting in recovery mode, when its configuration didn't load
- an add-on set to start on its own that isn't running
Repairs need an administrator's sign-in. Disk space and add-ons come from the Supervisor, so an install without one doesn't report them, and the page says so.
## Scheduled updates
Set a day and time under Scheduled updates on the page. It's off until you do. At that time Steward installs what's waiting, add-ons first, then Core, then the OS, one at a time and each after its own confirmed backup. It starts nothing new more than 4 hours after the slot. The first failure ends the run and the rest wait, and the page, the overview and `update_failed` say so. A Core update restarts Home Assistant with nobody there to watch, and an update can change how automations behave.
An OS update installed by the schedule also reboots the machine, with nobody there to watch. It counts as done once the new version is running. If that hasn't happened 20 minutes after the reboot, for example because the machine didn't come back or fell back to the old version, the update is marked failed.
## Metrics
Under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting`, `backup_age_hours`, `problems_open`, `disk_free_pct` and `update_failed`. Useful alert rules: `backup_age_hours` above 26 for a missed daily backup, `problems_open` above 0, and `update_failed` above 0. A rule's no-data warning covers Home Assistant going quiet.
+2
View File
@@ -46,6 +46,8 @@ Each target also records four metrics about itself, under its own name:
A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate. A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.
**A refused token stops the scraping.** When an endpoint answers 401 or 403, Steward stops asking and says so on the target's page and in the overview, because asking again won't change the answer and some services, Home Assistant among them, ban an address after a few refusals. Saving the target, usually with a new token, or pausing and resuming it, starts it again. Any other failure is retried at the next scrape.
A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state. A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state.
**Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were. **Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.
+22 -9
View File
@@ -49,9 +49,12 @@ def setup(app: "Quart") -> None:
from steward.core.norms import floor_setting, learn_normal from steward.core.norms import floor_setting, learn_normal
learn_normal("docker", "cpu_pct", floor_setting(app, "container_cpu_pct")) learn_normal("docker", "cpu_pct", floor_setting(app, "container_cpu_pct"))
learn_normal("docker", "mem_used_pct", floor_setting(app, "container_mem_pct")) learn_normal("docker", "mem_used_pct", floor_setting(app, "container_mem_pct"))
# A host's agent carries its containers' figures: they go silent with it (#5721). # A host's agent carries its containers' figures: they go silent with it
# (#5721). A swarm service's aren't tied to one host.
from steward.core.alerts import agent_fed from steward.core.alerts import agent_fed
agent_fed("docker", lambda resource: resource.split("/", 1)[0]) from .service import SERVICE_RESOURCE
agent_fed("docker", lambda resource: None if resource.startswith(SERVICE_RESOURCE)
else resource.split("/", 1)[0])
# An alert on a swarm figure names the services or nodes it counts (#6400). # An alert on a swarm figure names the services or nodes it counts (#6400).
from steward.core.alerts import names_behind from steward.core.alerts import names_behind
from .swarm_health import NAMERS from .swarm_health import NAMERS
@@ -63,9 +66,10 @@ def setup(app: "Quart") -> None:
def get_scheduled_tasks() -> list: def get_scheduled_tasks() -> list:
# Collection is now agent-driven (pushed to the host_agent ingest); the # Collection is agent-driven (pushed to the host_agent ingest). The one
# central socket scrape was removed. No periodic task to register. # task here moves task-named history to its service.
return [] from .rekey import make_rekey_task
return [make_rekey_task(_app)]
def get_blueprint(): def get_blueprint():
@@ -88,16 +92,25 @@ async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dic
return await events(db, host_id, host_name, since, until) return await events(db, host_id, host_name, since, until)
async def mcp_resource_events(db, source: str, resource: str, since, until) -> list[dict]:
from .service import SERVICE_RESOURCE
if source != "docker" or not resource.startswith(SERVICE_RESOURCE):
return []
from .mcp_tools import service_events
return await service_events(db, resource[len(SERVICE_RESOURCE):], since, until)
def get_nav() -> list[dict]: def get_nav() -> list[dict]:
# Surfaces in the sidebar's Infrastructure group (fleet-wide container view). # Surfaces in the sidebar's Infrastructure group (fleet-wide container view).
return [{"label": "Docker", "href": "/plugins/docker/"}] return [{"label": "Docker", "href": "/plugins/docker/"}]
def get_alert_metrics() -> dict[str, list[str]]: def get_alert_metrics() -> dict[str, list[str]]:
"""What ingest.py records per container (resource "<host>/<container>"): """What ingest.py records per container (resource "swarm/<service>" for a
restart_count is cumulative (alert on crash-looping); is_healthy is 1.0 / swarm task, else "<host>/<container>"): restart_count is cumulative (alert
0.0 from the container's HEALTHCHECK, only for containers that define one. on crash-looping); is_healthy is 1.0 / 0.0 from the container's HEALTHCHECK,
net_rx_bps / net_tx_bps are the bytes a second it receives and sends only for containers that define one. net_rx_bps / net_tx_bps are the bytes
a second it receives and sends, under "<host>/<service or container>"
(#6115): "a container flooding the uplink" is net_tx_bps over a level. (#6115): "a container flooding the uplink" is net_tx_bps over a level.
Per host (resource "<host>"), what Docker keeps on its disk (#5304). Per host (resource "<host>"), what Docker keeps on its disk (#5304).
Swarm-wide (resource "swarm"), from any manager's report: services short Swarm-wide (resource "swarm"), from any manager's report: services short
+9 -6
View File
@@ -404,6 +404,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
""" """
from steward.core.alerts import MetricBatch from steward.core.alerts import MetricBatch
from .models import DockerContainer, DockerEvent from .models import DockerContainer, DockerEvent
from .service import container_resource, traffic_resource
async def mark(name: str) -> None: async def mark(name: str) -> None:
if phases is not None: if phases is not None:
@@ -432,9 +433,11 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
latest_at, latest_containers = ordered[-1] latest_at, latest_containers = ordered[-1]
# Time series for every snapshot (running containers with a CPU read), in # Time series for every snapshot (running containers with a CPU read), in
# the one metric store (#5752): resource "<host>/<container>", so # the one metric store: resource "swarm/<service>" for a swarm task, so its
# containers of the same name on different hosts don't collide. Recorded # history survives the task being replaced or moved, else
# with the alert batch below, at each snapshot's own time. # "<host>/<container>", so containers of the same name on different hosts
# don't collide. Recorded with the alert batch below, at each snapshot's
# own time.
# Snapshot of stored per-container state BEFORE the upsert overwrites it — # Snapshot of stored per-container state BEFORE the upsert overwrites it —
# the network counters each rate counts from, and the lifecycle-event diff. # the network counters each rate counts from, and the lifecycle-event diff.
# Skip event derivation on the host's first snapshot (empty state) so we # Skip event derivation on the host's first snapshot (empty state) so we
@@ -448,7 +451,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
for c in containers: for c in containers:
if c.get("status") != "running" or c.get("cpu_pct") is None or not c.get("name"): if c.get("status") != "running" or c.get("cpu_pct") is None or not c.get("name"):
continue continue
resource = f"{host.name}/{c['name']}" resource = container_resource(host.name, c["name"])
for metric in ("cpu_pct", "mem_used_pct", "mem_used_bytes", "mem_cache_bytes"): for metric in ("cpu_pct", "mem_used_pct", "mem_used_bytes", "mem_cache_bytes"):
if c.get(metric) is not None: if c.get(metric) is not None:
batch.add(source_module="docker", resource_name=resource, batch.add(source_module="docker", resource_name=resource,
@@ -457,7 +460,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
# can be put down to the container behind it (#6115). # can be put down to the container behind it (#6115).
before = {r.name: (r.scraped_at, r.container_id, r.net_rx_bytes, r.net_tx_bytes) for r in old_rows} before = {r.name: (r.scraped_at, r.container_id, r.net_rx_bytes, r.net_tx_bytes) for r in old_rows}
for recorded_at, name, rx_bps, tx_bps in net_rates(before, ordered): for recorded_at, name, rx_bps, tx_bps in net_rates(before, ordered):
resource = f"{host.name}/{name}" resource = traffic_resource(host.name, name)
batch.add(source_module="docker", resource_name=resource, batch.add(source_module="docker", resource_name=resource,
metric_name="net_rx_bps", value=round(rx_bps, 1), recorded_at=recorded_at) metric_name="net_rx_bps", value=round(rx_bps, 1), recorded_at=recorded_at)
batch.add(source_module="docker", resource_name=resource, batch.add(source_module="docker", resource_name=resource,
@@ -524,7 +527,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
existing.blk_read_bytes = c.get("blk_read_bytes") existing.blk_read_bytes = c.get("blk_read_bytes")
existing.blk_write_bytes = c.get("blk_write_bytes") existing.blk_write_bytes = c.get("blk_write_bytes")
resource = f"{host.name}/{name}" resource = container_resource(host.name, name)
# Restart count is alertable regardless of state (crash-looping matters # Restart count is alertable regardless of state (crash-looping matters
# most while the container is down/restarting). # most while the container is down/restarting).
batch.add( batch.add(
+15
View File
@@ -772,6 +772,21 @@ async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dic
subject=f"{host_name}/{e.container_name}") for e in rows] subject=f"{host_name}/{e.container_name}") for e in rows]
async def service_events(db, service: str, since, until) -> list[dict]:
"""What happened to a swarm service's containers on every host in a window
(get_anomaly): starts with their image, deaths, OOM kills, health changes.
A service has no one host to read events by."""
from steward.mcp.tools._events import event
from steward.models.hosts import Host
rows = (await db.execute(
select(DockerEvent, Host.name).join(Host, Host.id == DockerEvent.host_id)
.where(DockerEvent.container_name.startswith(f"{service}.", autoescape=True),
DockerEvent.at >= since, DockerEvent.at <= until)
.order_by(DockerEvent.at).limit(500))).all()
return [event(e.at, "container", f"{e.container_name} {e.event}" + (f" ({e.detail})" if e.detail else ""),
subject=f"{host}/{e.container_name}") for e, host in rows]
def register_tools(register) -> None: def register_tools(register) -> None:
for fn in (list_containers, get_container, list_swarm_services, for fn in (list_containers, get_container, list_swarm_services,
search_container_logs, get_docker_disk_usage): search_container_logs, get_docker_disk_usage):
+198
View File
@@ -0,0 +1,198 @@
# plugins/docker/rekey.py
"""Moves history kept under a swarm task's own name, "<host>/<service>.<slot>.
<task>", onto its service's key, "swarm/<service>": samples, hourly rollups,
anomalies and the alert rules and maintenance windows that name it.
The raw table holds millions of rows, too many to move in a migration that the
app has to wait for. A small task moves what it can in a short pass in the
background instead, newest history first, and does nothing once none are left. Network
rates stay where they are: they are kept per host, and age out with retention.
"""
from __future__ import annotations
import asyncio
import logging
import time
from collections import Counter
from typing import TYPE_CHECKING
from sqlalchemy import delete, select, text, update
from steward.core.scheduler import ScheduledTask
from steward.models.alerts import AlertRule, MaintenanceWindow
from steward.models.metrics import PluginMetric, PluginMetricHourly, PluginMetricLatest
from steward.models.norms import MetricAnomaly, MetricAnomalySpan
from steward.models.tickets import TicketLink
from steward.models.timeline import TimelineEntry
from steward.timeline.record import anomaly_changed
from .service import container_resource, task_service
if TYPE_CHECKING:
from quart import Quart
logger = logging.getLogger(__name__)
BATCH = 1000 # raw samples moved in one go: each rewrites its row and every index on it
EVERY_S = 15
PASS_S = 2 # a key with only a few rows takes milliseconds, and there can be many
REST = 4 # after a pass, wait this many times as long as it took, so ingest keeps the disk
TRAFFIC = ("net_rx_bps", "net_tx_bps")
_TASK_KEY = r"^[^/]+/.+\.([0-9]+|[a-z0-9]{25})\.[a-z0-9]{25}$"
_NOT_TRAFFIC = "metric_name NOT IN ('net_rx_bps', 'net_tx_bps')"
# Hours the service has no row for change name; hours it has are combined, as
# two tasks of a service can share an hour.
_RENAME_HOURS = text(f"""
UPDATE plugin_metrics_hourly o SET resource_name = :new
WHERE o.source_module = 'docker' AND o.resource_name = :old AND o.{_NOT_TRAFFIC}
AND NOT EXISTS (SELECT 1 FROM plugin_metrics_hourly n
WHERE n.source_module = 'docker' AND n.resource_name = :new
AND n.metric_name = o.metric_name AND n.bucket = o.bucket)
""")
_COMBINE_HOURS = text("""
UPDATE plugin_metrics_hourly n SET
value_avg = COALESCE((n.value_avg * n.sample_count + o.value_avg * o.sample_count)
/ NULLIF(n.sample_count + o.sample_count, 0), n.value_avg),
value_max = GREATEST(n.value_max, o.value_max),
value_p95 = GREATEST(n.value_p95, o.value_p95),
sample_count = n.sample_count + o.sample_count
FROM plugin_metrics_hourly o
WHERE o.source_module = 'docker' AND o.resource_name = :old
AND n.source_module = 'docker' AND n.resource_name = :new
AND n.metric_name = o.metric_name AND n.bucket = o.bucket
""")
_DROP_HOURS = text(f"""
DELETE FROM plugin_metrics_hourly
WHERE source_module = 'docker' AND resource_name = :old AND {_NOT_TRAFFIC}
""")
def service_key(old: str) -> str:
"""The key a task-named resource belongs under now."""
return container_resource("", old.split("/", 1)[1])
async def old_keys(session) -> list[str]:
"""Every task-named docker resource still holding something to move, to be
taken from the end: those with raw samples last, as they are what a chart
and a normal read first. The latest-value table stands for the raw samples,
as a key stays in it until its samples have aged out."""
async def names(column, *where) -> set[str]:
found = (await session.execute(select(column).distinct().where(*where, column.op("~")(_TASK_KEY)))).scalars()
return {k for k in found if task_service(k.split("/", 1)[1])}
recent = await names(PluginMetricLatest.resource_name, PluginMetricLatest.source_module == "docker",
PluginMetricLatest.metric_name.notin_(TRAFFIC))
hourly = await names(PluginMetricHourly.resource_name, PluginMetricHourly.source_module == "docker",
PluginMetricHourly.metric_name.notin_(TRAFFIC))
named = (await names(MetricAnomaly.resource_name, MetricAnomaly.source_module == "docker")
| await names(AlertRule.resource_name, AlertRule.source_module == "docker")
| await names(MaintenanceWindow.scope_resource, MaintenanceWindow.scope_source == "docker"))
logger.info("Swarm history under task names: %d keys with raw samples, %d with hourly rows, "
"%d named by anomalies, rules or windows", len(recent), len(hourly), len(named))
by_service = Counter(task_service(k.split("/", 1)[1]) for k in hourly - recent)
logger.info("Swarm history under task names, hourly only, most keys first: %s; for example %s",
by_service.most_common(10), sorted(hourly - recent)[:3])
return sorted(hourly - recent) + sorted((named - hourly) - recent) + sorted(recent)
async def _merge_anomalies(session, new: str) -> None:
"""One row per metric still to review, as for any key: the rows waiting on
the service become the newest of them, with all their occurrences."""
rows = (await session.execute(
select(MetricAnomaly).where(
MetricAnomaly.source_module == "docker", MetricAnomaly.resource_name == new,
MetricAnomaly.review == "to_review")
.order_by(MetricAnomaly.last_started_at.desc()))).scalars().all()
keep: dict[str, MetricAnomaly] = {}
for row in rows:
into = keep.setdefault(row.metric_name, row)
if into is row:
continue
await session.execute(update(MetricAnomalySpan).where(
MetricAnomalySpan.anomaly_id == row.id).values(anomaly_id=into.id))
into.occurrences += row.occurrences
into.started_at = min(into.started_at, row.started_at)
if row.ended_at is None:
into.ended_at = None
if row.peak_level > into.peak_level:
into.peak_level, into.peak_at = row.peak_level, row.peak_at
into.usual_hold, into.usual_peak = row.usual_hold, row.usual_peak
linked = (await session.execute(select(TicketLink.ticket_id).where(
TicketLink.ref_type == "metric_anomaly", TicketLink.ref_id == into.id))).scalars().all()
await session.execute(update(TicketLink).where(
TicketLink.ref_type == "metric_anomaly", TicketLink.ref_id == row.id,
TicketLink.ticket_id.notin_(linked)).values(ref_id=into.id))
await session.execute(delete(TicketLink).where(
TicketLink.ref_type == "metric_anomaly", TicketLink.ref_id == row.id))
await session.execute(delete(TimelineEntry).where(
TimelineEntry.ref_type == "metric_anomaly", TimelineEntry.ref_id == row.id))
await session.delete(row)
async def move_key(session, old: str) -> bool:
"""Move the next batch of `old`, and the rest of what it holds once its raw
samples are done. Returns whether it is finished. Runs inside the caller's
transaction."""
new = service_key(old)
batch = select(PluginMetric.id).where(
PluginMetric.source_module == "docker", PluginMetric.resource_name == old,
PluginMetric.metric_name.notin_(TRAFFIC)).limit(BATCH)
moved = await session.execute(
update(PluginMetric).where(PluginMetric.id.in_(batch)).values(resource_name=new)
.execution_options(synchronize_session=False))
if moved.rowcount >= BATCH:
return False
keys = {"old": old, "new": new}
for statement in (_RENAME_HOURS, _COMBINE_HOURS, _DROP_HOURS):
await session.execute(statement, keys)
await session.execute(delete(PluginMetricLatest).where(
PluginMetricLatest.source_module == "docker", PluginMetricLatest.resource_name == old,
PluginMetricLatest.metric_name.notin_(TRAFFIC)))
await session.execute(update(AlertRule).where(
AlertRule.source_module == "docker", AlertRule.resource_name == old).values(resource_name=new))
await session.execute(update(MaintenanceWindow).where(
MaintenanceWindow.scope_source == "docker", MaintenanceWindow.scope_resource == old)
.values(scope_resource=new))
renamed = await session.execute(update(MetricAnomaly).where(
MetricAnomaly.source_module == "docker", MetricAnomaly.resource_name == old)
.values(resource_name=new).execution_options(synchronize_session=False))
if not renamed.rowcount:
return True
session.expire_all()
await _merge_anomalies(session, new)
await session.flush()
for row in (await session.execute(select(MetricAnomaly).where(
MetricAnomaly.source_module == "docker", MetricAnomaly.resource_name == new))).scalars().all():
await anomaly_changed(session, row)
return True
def make_rekey_task(app: "Quart") -> ScheduledTask:
remaining: list[str] | None = None
busy = asyncio.Lock()
resume_at = 0.0
async def rekey() -> None:
nonlocal remaining, resume_at
started = time.monotonic()
if remaining == [] or busy.locked() or started < resume_at:
return
async with busy, app.db_sessionmaker() as session:
if remaining is None:
async with session.begin():
remaining = await old_keys(session)
deadline = time.monotonic() + PASS_S
while remaining and time.monotonic() < deadline:
async with session.begin():
finished = await move_key(session, remaining[-1])
if not finished: # a full batch of samples is enough for one pass
break
remaining.pop()
if not remaining:
logger.info("Swarm history under task names: all moved")
now = time.monotonic()
resume_at = now + REST * (now - started)
return ScheduledTask(name="docker_rekey", coro_factory=rekey, interval_seconds=EVERY_S)
+1 -1
View File
@@ -482,7 +482,7 @@ async def container_detail(host_id: str, name: str):
resource = docker_svc.container_resource(host.name, name) if host is not None else None resource = docker_svc.container_resource(host.name, name) if host is not None else None
anomalies = await anom.views(db, await anom.list_anomalies( anomalies = await anom.views(db, await anom.list_anomalies(
db, source="docker", resource=resource, db, source="docker", resource=resource,
since=datetime.now(timezone.utc) - timedelta(days=30), limit=20), since=datetime.now(timezone.utc) - timedelta(days=30), limit=100),
links=False) if resource else [] links=False) if resource else []
if c is None: if c is None:
+38 -11
View File
@@ -227,7 +227,8 @@ async def container_norms(db, keys, now: datetime | None = None) -> dict:
now or datetime.now(timezone.utc)) now or datetime.now(timezone.utc))
out: dict = {} out: dict = {}
for (_s, resource, metric), norm in got.items(): for (_s, resource, metric), norm in got.items():
out.setdefault(resources[resource], {})[metric] = norm for key in resources[resource]:
out.setdefault(key, {})[metric] = norm
return out return out
@@ -318,18 +319,43 @@ async def query_logs(db, host_id: str, name: str, stream: str, query: str,
CONTAINER_SERIES = ("cpu_pct", "mem_used_pct", "mem_used_bytes", "mem_cache_bytes") CONTAINER_SERIES = ("cpu_pct", "mem_used_pct", "mem_used_bytes", "mem_cache_bytes")
# A container's series live in the one metric store (#5752): source "docker", # A swarm task's container: "<service>.<slot or node id>.<task id>".
# resource "<host name>/<container name>", recorded at ingest. _TASK = re.compile(r"^(?P<service>.+?)\.(?:\d+|[a-z0-9]{25})\.[a-z0-9]{25}$")
SERVICE_RESOURCE = "swarm/"
def task_service(name: str) -> str | None:
"""The swarm service a container is a task of, from its name, or None."""
m = _TASK.match(name or "")
return m["service"] if m else None
# A container's series live in the one metric store: source "docker", recorded
# at ingest. A swarm service's are under "swarm/<service>", which stays the same
# when a task is replaced or moves host, and which all its tasks share. Any
# other container's are under "<host name>/<container name>".
def container_resource(host_name: str, name: str) -> str: def container_resource(host_name: str, name: str) -> str:
return f"{host_name}/{name}" service = task_service(name)
return f"{SERVICE_RESOURCE}{service}" if service else f"{host_name}/{name}"
async def container_resources(db, keys) -> dict[str, tuple[str, str]]: def traffic_resource(host_name: str, name: str) -> str:
"""{resource: (host_id, container_name)} for (host_id, container_name) """Where a container's network rates are kept: "<host name>/<service or
keys whose host still exists.""" container>", as they say who on that host is carrying its traffic."""
return f"{host_name}/{task_service(name) or name}"
async def container_resources(db, keys) -> dict[str, list[tuple[str, str]]]:
"""{resource: [(host_id, container_name), ...]} for (host_id,
container_name) keys whose host still exists. A swarm service's tasks share
one resource."""
keys = set(keys) keys = set(keys)
hosts = await host_map(db, {h for h, _ in keys}) hosts = await host_map(db, {h for h, _ in keys})
return {container_resource(hosts[h].name, n): (h, n) for h, n in keys if h in hosts} out: dict[str, list[tuple[str, str]]] = {}
for h, n in keys:
if h in hosts:
out.setdefault(container_resource(hosts[h].name, n), []).append((h, n))
return out
async def container_points(db, host_id: str, name: str, metric: str, since, async def container_points(db, host_id: str, name: str, metric: str, since,
@@ -376,7 +402,7 @@ async def average_cpu(db, keys, since: datetime) -> dict[tuple[str, str], float]
PluginMetric.metric_name == "cpu_pct", PluginMetric.metric_name == "cpu_pct",
PluginMetric.recorded_at >= since) PluginMetric.recorded_at >= since)
.group_by(PluginMetric.resource_name))).all() .group_by(PluginMetric.resource_name))).all()
return {resources[r]: float(v) for r, v in rows if v is not None} return {key: float(v) for r, v in rows if v is not None for key in resources[r]}
# Busiest Containers bars (#5214): the recent past in BARS equal slices. # Busiest Containers bars (#5214): the recent past in BARS equal slices.
@@ -402,9 +428,10 @@ async def recent_usage(db, keys, since: datetime, width: int) -> dict:
out: dict = {} out: dict = {}
for resource, at, metric, value in await _samples( for resource, at, metric, value in await _samples(
db, resources, ("cpu_pct", "mem_used_bytes"), since, width): db, resources, ("cpu_pct", "mem_used_bytes"), since, width):
cpu, mem = out.setdefault(resources[resource], {}).get(at, (None, None))
v = None if value is None else float(value) v = None if value is None else float(value)
out[resources[resource]][at] = (v, mem) if metric == "cpu_pct" else (cpu, v) for key in resources[resource]:
cpu, mem = out.setdefault(key, {}).get(at, (None, None))
out[key][at] = (v, mem) if metric == "cpu_pct" else (cpu, v)
return out return out
+46
View File
@@ -0,0 +1,46 @@
# plugins/homeassistant/__init__.py
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from quart import Quart
_app: "Quart | None" = None
def setup(app: "Quart") -> None:
global _app
_app = app
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate # noqa: F401 — registers the tables
from .timeline import register
register()
def get_scheduled_tasks() -> list:
from .scheduler import make_tick_task
return [make_tick_task(_app)]
def get_blueprint():
from .routes import homeassistant_bp
return homeassistant_bp
def get_nav() -> list[dict]:
return [{"label": "Home Assistant", "href": "/plugins/homeassistant/"}]
def get_alert_metrics() -> dict[str, list[str]]:
"""What the poll records, all under the resource "homeassistant"."""
return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting", "problems_open", "disk_free_pct",
"update_failed"]}
def register_mcp_tools(register) -> None:
from .mcp_tools import register_tools
register_tools(register)
async def mcp_overview(db) -> list[dict]:
from .mcp_tools import mcp_overview as findings
return await findings(db)
+292
View File
@@ -0,0 +1,292 @@
# plugins/homeassistant/client.py
"""Talking to Home Assistant, with what its own sign-in gave Steward: a refresh
token kept in the connection, and access tokens made from it that last half an
hour. Repairs and the Supervisor's reports are only offered on its websocket. Home Assistant bans an address after a few refused requests, so nothing is
sent without a token and a refusal stops the asking. Every failure comes back as
one sentence a page can show."""
from __future__ import annotations
import asyncio
import json
import ssl
import time
from contextlib import asynccontextmanager
from urllib.parse import urlencode
import httpx
from websockets.asyncio.client import connect as ws_connect
from websockets.exceptions import WebSocketException
from steward.core.crypto import decrypt_secret
TIMEOUT_SECONDS = 15.0
# Home Assistant answers an action when it has finished, minutes later for a
# backup or an update, and carries on if the request ends. So its answer is only
# waited for this long, and the outcome is read from its entities.
ACTION_READ_SECONDS = 10.0
MAX_BYTES = 16 * 1024 * 1024
# An access token this close to running out isn't used for another request.
EXPIRY_MARGIN_SECONDS = 60
# One read of the system over the websocket, however its answers dribble in.
SYSTEM_READ_SECONDS = 30
_access: dict[str, tuple[str, float]] = {} # refresh token -> access token, when to stop using it
class Unreachable(Exception):
"""Home Assistant didn't answer usefully; it may be restarting."""
class Refused(Exception):
"""Home Assistant refused Steward's sign-in, or has banned this address."""
class Rejected(Exception):
"""Home Assistant understood an action and wouldn't do it."""
def _refusal(status: int) -> Refused | None:
if status == 401:
_access.clear()
return Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.")
if status == 403:
return Refused("Home Assistant answered 403: it may have banned Steward's address "
"after refused requests. Look in its ip_bans.yaml.")
return None
async def _request(method: str, url: str, path: str, verify_tls: bool, transport, *, token: str | None = None,
read_timeout: float = TIMEOUT_SECONDS, **kwargs) -> tuple[int, bytes]:
headers = {"Authorization": f"Bearer {token}"} if token else {}
timeout = httpx.Timeout(TIMEOUT_SECONDS, read=read_timeout)
try:
# A slow drip of bytes can keep each read inside its own timeout for ever.
async with asyncio.timeout(TIMEOUT_SECONDS + read_timeout):
async with httpx.AsyncClient(timeout=timeout, verify=verify_tls, transport=transport,
follow_redirects=False) as client:
async with client.stream(method, url.rstrip("/") + path, headers=headers, **kwargs) as response:
body = bytearray()
async for chunk in response.aiter_bytes():
body.extend(chunk)
if len(body) > MAX_BYTES:
raise Unreachable("Home Assistant's answer was too large to read.")
return response.status_code, bytes(body)
except TimeoutError:
raise httpx.ReadTimeout("no complete answer in time") from None
def _unreachable(exc: httpx.HTTPError) -> Unreachable:
if isinstance(exc, httpx.TimeoutException):
return Unreachable(f"Home Assistant didn't answer within {TIMEOUT_SECONDS:.0f} s.")
return Unreachable(f"Couldn't reach Home Assistant: {exc}")
# ── Signing in ───────────────────────────────────────────────────────────────
def sign_in_url(url: str, client_id: str, redirect_uri: str, state: str) -> str:
"""Home Assistant's own sign-in page, which sends the browser back to
`redirect_uri`. It accepts `client_id` as it is because the two share an
address, so it needn't be able to reach Steward."""
return f"{url.rstrip('/')}/auth/authorize?" + urlencode(
{"response_type": "code", "client_id": client_id, "redirect_uri": redirect_uri, "state": state})
async def _tokens(url: str, form: dict, verify_tls: bool, transport) -> dict:
try:
status, body = await _request("POST", url, "/auth/token", verify_tls, transport, data=form)
except httpx.HTTPError as exc:
raise _unreachable(exc) from None
if status in (400, 401, 403):
raise Refused("Home Assistant no longer accepts Steward's sign-in. Sign in again.")
if status != 200:
raise Unreachable(f"Home Assistant answered {status}.")
try:
found = json.loads(body)
found["access_token"], int(found["expires_in"])
except (ValueError, KeyError, TypeError):
raise Unreachable("Home Assistant's answer wasn't a sign-in.") from None
return found
def _remember(refresh_token: str, found: dict) -> str:
until = time.monotonic() + int(found["expires_in"]) - EXPIRY_MARGIN_SECONDS
_access[refresh_token] = (found["access_token"], until)
return found["access_token"]
async def redeem_code(url: str, client_id: str, code: str, verify_tls: bool, transport=None) -> tuple[str, str]:
"""(refresh token, access token) for the code the sign-in page sent back."""
found = await _tokens(url, {"grant_type": "authorization_code", "client_id": client_id, "code": code},
verify_tls, transport)
refresh_token = found.get("refresh_token")
if not refresh_token:
raise Unreachable("Home Assistant's answer wasn't a sign-in.")
return refresh_token, _remember(refresh_token, found)
async def access_token(connection, transport=None) -> str:
refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in")
token, until = _access.get(refresh_token, (None, 0.0))
if token and time.monotonic() < until:
return token
found = await _tokens(connection.url, {"grant_type": "refresh_token", "client_id": connection.client_id,
"refresh_token": refresh_token}, connection.verify_tls, transport)
return _remember(refresh_token, found)
async def is_admin(url: str, token: str, verify_tls: bool, transport=None) -> bool:
"""Whether the signed-in user is an administrator, which its API only says by
refusing someone who isn't. So this is asked once, right after signing in, and
a refusal here counts once toward a ban."""
try:
status, _ = await _request("POST", url, "/api/template", verify_tls, transport, token=token,
json={"template": "1"})
except httpx.HTTPError as exc:
raise _unreachable(exc) from None
if status == 401:
return False
if status != 200:
raise (_refusal(status) or Unreachable(f"Home Assistant answered {status}."))
return True
async def revoke(connection, transport=None) -> None:
"""End this sign-in on Home Assistant. It answers 200 whether or not it knew it."""
refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in")
_access.pop(refresh_token, None)
try:
status, _ = await _request("POST", connection.url, "/auth/revoke", connection.verify_tls, transport,
data={"token": refresh_token})
except httpx.HTTPError as exc:
raise _unreachable(exc) from None
if status != 200:
raise Unreachable(f"Home Assistant answered {status}.")
# ── Reading and acting ───────────────────────────────────────────────────────
async def _get(url: str, path: str, token: str, verify_tls: bool, transport) -> list | dict | None:
"""The parsed JSON of a GET, or None for a 404."""
try:
status, body = await _request("GET", url, path, verify_tls, transport, token=token)
except httpx.HTTPError as exc:
raise _unreachable(exc) from None
if refusal := _refusal(status):
raise refusal
if status == 404:
return None
if status != 200:
raise Unreachable(f"Home Assistant answered {status}.")
try:
return json.loads(body)
except ValueError:
raise Unreachable("Home Assistant's answer wasn't JSON.") from None
async def read_states(url: str, token: str, verify_tls: bool, transport=None) -> list[dict]:
states = await _get(url, "/api/states", token, verify_tls, transport)
if not isinstance(states, list):
raise Unreachable("Home Assistant's answer wasn't a list of entities.")
return states
async def read_state(url: str, entity_id: str, token: str, verify_tls: bool, transport=None) -> dict | None:
return await _get(url, f"/api/states/{entity_id}", token, verify_tls, transport)
async def call_action(url: str, domain: str, action: str, data: dict, token: str, verify_tls: bool,
transport=None) -> None:
"""Ask Home Assistant to do something. Returning means it got the request,
not that it has finished: a request cut off once sent has been taken up, and
an update that restarts Home Assistant cuts its own request off."""
try:
status, body = await _request("POST", url, f"/api/services/{domain}/{action}", verify_tls, transport,
token=token, read_timeout=ACTION_READ_SECONDS, json=data)
except (httpx.ReadTimeout, httpx.ReadError, httpx.RemoteProtocolError):
return
except httpx.HTTPError as exc:
raise _unreachable(exc) from None
if refusal := _refusal(status):
raise refusal
if status != 200:
try:
said = json.loads(body).get("message")
except (ValueError, AttributeError):
said = None
raise Rejected(said or f"Home Assistant answered {status}.")
# ── The system, over the websocket ───────────────────────────────────────────
class _Socket:
"""Home Assistant's websocket after sign-in: one question at a time, each with
an id, and None for one it won't answer (no Supervisor, or not an administrator)."""
def __init__(self, ws):
self.ws, self.asked = ws, 0
async def sign_in(self, token: str) -> None:
await self.ws.recv()
await self.ws.send(json.dumps({"type": "auth", "access_token": token}))
if json.loads(await self.ws.recv()).get("type") != "auth_ok":
raise Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.")
async def ask(self, **question):
self.asked += 1
await self.ws.send(json.dumps({"id": self.asked, **question}))
while True:
answer = json.loads(await self.ws.recv())
if answer.get("id") == self.asked:
return answer.get("result") if answer.get("success") else None
async def supervisor(self, endpoint: str):
return await self.ask(type="supervisor/api", endpoint=endpoint, method="get")
@asynccontextmanager
async def _signed_in(url: str, token: str, verify_tls: bool, connect):
"""A signed-in socket. `connect` stands in for the websocket in tests."""
target = "ws" + url.rstrip("/").removeprefix("http") + "/api/websocket"
options: dict = {"max_size": MAX_BYTES}
if target.startswith("wss:") and not verify_tls:
options["ssl"] = ssl.create_default_context()
options["ssl"].check_hostname, options["ssl"].verify_mode = False, ssl.CERT_NONE
try:
async with asyncio.timeout(SYSTEM_READ_SECONDS):
async with (connect or ws_connect)(target, **options) as ws:
socket = _Socket(ws)
await socket.sign_in(token)
yield socket
except TimeoutError:
raise Unreachable(f"Home Assistant's system report didn't arrive within {SYSTEM_READ_SECONDS} s.") from None
except (OSError, ValueError, KeyError, WebSocketException) as exc:
raise Unreachable(f"Couldn't read Home Assistant's system report: {exc}") from None
async def read_system(url: str, token: str, verify_tls: bool, connect=None) -> dict:
"""What service.read_health needs, as Home Assistant answered it."""
async with _signed_in(url, token, verify_tls, connect) as socket:
found = {"config": await socket.ask(type="get_config"),
"issues": (await socket.ask(type="repairs/list_issues") or {}).get("issues"),
"resolution": await socket.supervisor("/resolution/info"),
"host": await socket.supervisor("/host/info"), "titles": {}, "boots": {}}
domains = sorted({i["domain"] for i in found["issues"] or []})
if domains:
titles = await socket.ask(type="frontend/get_translations", language="en",
category="issues", integration=domains)
found["titles"] = (titles or {}).get("resources") or {}
found["addons"] = (await socket.supervisor("/addons") or {}).get("addons")
# Whether one should be running is only on its own page, so a running one isn't asked.
for addon in found["addons"] or []:
if addon.get("state") != "started":
info = await socket.supervisor(f"/addons/{addon['slug']}/info") or {}
found["boots"][addon["slug"]] = info.get("boot")
return found
async def read_os(url: str, token: str, verify_tls: bool, connect=None) -> dict | None:
"""The Supervisor's report on the OS: `version`, and `version_pending` while a
written update waits for its reboot. None if it isn't answering yet, as just
after a boot."""
async with _signed_in(url, token, verify_tls, connect) as socket:
return await socket.supervisor("/os/info")
+168
View File
@@ -0,0 +1,168 @@
"""The Home Assistant plugin's MCP tools and its get_overview findings.
get_home_assistant reads what the poll stored. update_home_assistant starts an
update as the page's Install does. The sign-in Steward holds is never returned.
"""
from __future__ import annotations
from datetime import timezone
from sqlalchemy import select
from steward.core import weekly
from steward.core.formatting import zone
from steward.mcp._context import current_caller, mcp_app
from steward.mcp.tools._audit import audit_mcp, mcp_username
from steward.mcp.tools._shape import age_s, db_session, iso, now_utc, redact_url
from steward.mcp.tools.overview import finding
from steward.models.users import UserRole
from . import schedule
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
from .routes import connection_says
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
from .updates import CannotStart, start_update
RECENT_UPDATES = 10
def _schedule_view(plan, now) -> dict | None:
if plan is None:
return None
return {"enabled": plan.enabled, "when": weekly.describe(plan),
"next_slot_at": iso(weekly.next_slot(plan, now)) if plan.enabled else None,
"last_slot_at": iso(weekly.fired_slot(plan)), "stopped": plan.stopped}
def _run_view(r: HomeAssistantUpdate) -> dict:
return {"update_id": r.id, "entity_id": r.entity_id, "title": r.title, "kind": r.kind,
"from_version": r.from_version, "to_version": r.to_version, "state": r.state, "message": r.message,
"started_by": r.started_by, "started_at": iso(r.started_at), "backup_at": iso(r.backup_at),
"finished_at": iso(r.finished_at)}
async def get_home_assistant() -> dict:
"""Home Assistant as of the last poll: whether it answers, which updates are
waiting, how old its last backup is, and the updates Steward was asked to install.
`installable` is what update_home_assistant takes (Core, OS, add-ons);
`others_waiting` is firmware and custom integrations, which Steward leaves
to Home Assistant. `can_install` is whether the Home Assistant user who signed
Steward in is an administrator, which installing needs. `system.problems` is
what's wrong with the system itself: open repairs, Supervisor health, low disk,
recovery mode, add-ons that should be running. `schedule` is the weekly slot
for installing what's waiting, null until one is set.
"""
now = now_utc()
async with db_session() as db:
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
if c is None:
return {"connected": False, "says": "No connection yet: an admin signs in on the Home Assistant page."}
reach, says = connection_says(c, now)
backup = c.backup or {}
age = backup_age_hours(backup, now)
return {"connected": True, "url": redact_url(c.url), "reach": reach, "says": says,
"polled_at": iso(c.last_polled_at), "ok_age_s": age_s(c.last_ok_at, now), "can_install": c.can_install,
"installable": [{k: u[k] for k in ("entity_id", "title", "kind", "installed", "latest", "in_progress")}
for u in installable(c)],
"others_waiting": [u["title"] for u in c.updates or [] if u["waiting"] and not u["kind"]],
"backup": {"verdict": backup_verdict(backup, now)[1], "last_ok": iso(parse_moment(backup.get("last_ok"))),
"age_hours": round(age, 1) if age is not None else None, "stale_after_hours": STALE_BACKUP_HOURS,
"next": iso(parse_moment(backup.get("next"))), "last_result": backup.get("last_result")},
"system": c.health or {}, "schedule": _schedule_view(plan, now),
"updates": [_run_view(r) for r in runs]}
async def update_home_assistant(entity_id: str) -> dict:
"""Install one Core, OS or add-on update, as the page's Install does: Steward
takes a backup, confirms it landed, installs, and follows it to its end.
entity_id: from get_home_assistant's `installable`.
An update restarts Home Assistant or the add-on. An OS update reads as done
once it's written, and takes effect after a reboot, which Steward leaves to a
person. Refused while another is under way, or if nothing is waiting there.
Admin only, as on the page. Recorded in the audit log.
"""
caller = current_caller()
try:
run = await start_update(mcp_app(), entity_id, mcp_username(caller.username))
except CannotStart as exc:
raise ValueError(f"Nothing was started: {exc}.") from None
await audit_mcp("homeassistant.update_started", "homeassistant_update", run.id,
{"entity_id": entity_id, "title": run.title, "to": run.to_version})
return {**_run_view(run), "next": "get_home_assistant follows it: `updates` shows its state."}
async def set_home_assistant_schedule(enabled: bool, weekday: int = 5, at_time: str = "05:00",
time_zone: str = "UTC") -> dict:
"""Set the weekly slot for installing what's waiting, or turn it off, as the page does.
weekday: 0 for Monday to 6 for Sunday. at_time: "HH:MM" in time_zone.
At each slot Steward installs add-ons, then Core, then the OS, one at a time,
each after a confirmed backup, stopping at the first failure. A Core update
restarts Home Assistant and an OS update reboots the machine, which Steward
does itself here, with nobody watching. The first slot is the next one from
now. Admin only, and audited.
"""
try:
weekly.parse_at(at_time)
except ValueError:
raise ValueError(f"at_time is HH:MM, not {at_time!r}.") from None
if not 0 <= weekday <= 6:
raise ValueError("weekday is 0 for Monday to 6 for Sunday.")
if zone(time_zone) is timezone.utc and time_zone.upper() != "UTC":
raise ValueError(f"Unknown time zone {time_zone!r}.")
row = await schedule.save(mcp_app(), enabled=enabled, weekday=weekday, at_time=at_time, time_zone=time_zone)
await audit_mcp("homeassistant.schedule_saved", "homeassistant_schedule", row.id,
{"enabled": row.enabled, "when": weekly.describe(row)})
return _schedule_view(row, now_utc())
def overview_findings(c, latest_run, now, plan=None) -> list[dict]:
"""A connection that isn't answering, a problem with the system, stale backups,
a newest update that failed, and a scheduled run that stopped. Updates waiting
aren't a problem to raise."""
if c is None:
return []
found = []
reach, says = connection_says(c, now)
if reach == "down":
found.append(finding("warning", "home_assistant_down", "Home Assistant",
f"Home Assistant isn't answering: {says}", since=c.last_ok_at, next="get_home_assistant()"))
problems = problems_open(c.health)
if problems:
more = f" (and {len(problems) - 1} more)" if len(problems) > 1 else ""
found.append(finding("warning", "home_assistant_problem", "Home Assistant", f"{problems[0]['title']}{more}",
next="get_home_assistant()"))
tone, backup_says = backup_verdict(c.backup, now)
if c.last_ok_at is not None and tone == "warn":
found.append(finding("warning", "home_assistant_backup", "Home Assistant", backup_says,
since=parse_moment((c.backup or {}).get("last_ok")), next="get_home_assistant()"))
if latest_run is not None and latest_run.state == "failed":
found.append(finding("warning", "home_assistant_update_failed", latest_run.title,
f"The update of {latest_run.title} failed: {latest_run.message}",
since=latest_run.finished_at, next="get_home_assistant()"))
if plan is not None and plan.stopped:
found.append(finding("warning", "home_assistant_schedule_stopped", "Home Assistant",
f"The scheduled update run stopped: {plan.stopped}", since=weekly.fired_slot(plan),
next="get_home_assistant()"))
return found
async def mcp_overview(db) -> list[dict]:
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
latest = (await db.execute(select(HomeAssistantUpdate).order_by(
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
return overview_findings(c, latest, now_utc(), plan)
def register_tools(register) -> None:
register(get_home_assistant, role=UserRole.viewer, read_only=True)
register(update_home_assistant, role=UserRole.admin, read_only=False)
register(set_home_assistant_schedule, role=UserRole.admin, read_only=False)
@@ -0,0 +1,59 @@
"""Home Assistant: the connection and the updates Steward was asked to install
Revision ID: homeassistant_001_connection
Revises: (none — a branch off core via depends_on)
Create Date: 2026-10-31
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "homeassistant_001_connection"
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = "homeassistant"
depends_on: Union[str, Sequence[str], None] = ("0004_app_settings",)
def upgrade() -> None:
op.create_table(
"homeassistant_connection",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("url", sa.String(500), nullable=False),
sa.Column("token", sa.Text, nullable=False),
sa.Column("webhook_id", sa.Text, nullable=False),
sa.Column("verify_tls", sa.Boolean, nullable=False, server_default=sa.true()),
sa.Column("refused", sa.Boolean, nullable=False, server_default=sa.false()),
sa.Column("last_polled_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_ok_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error", sa.Text, nullable=True),
sa.Column("updates", sa.JSON, nullable=False, server_default="[]"),
sa.Column("backup", sa.JSON, nullable=False, server_default="{}"),
sa.Column("automation", sa.String(20), nullable=True),
)
op.create_table(
"homeassistant_updates",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("entity_id", sa.String(255), nullable=False),
sa.Column("title", sa.String(255), nullable=False),
sa.Column("kind", sa.String(10), nullable=False),
sa.Column("from_version", sa.String(100), nullable=True),
sa.Column("to_version", sa.String(100), nullable=True),
sa.Column("state", sa.String(20), nullable=False, server_default="backing_up"),
sa.Column("message", sa.Text, nullable=True),
sa.Column("started_by", sa.String(100), nullable=False, server_default=""),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("backup_before", sa.String(64), nullable=True),
sa.Column("backup_event_before", sa.String(64), nullable=True),
sa.Column("triggered_before", sa.String(64), nullable=True),
sa.Column("backup_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("install_sent_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
)
op.execute("CREATE UNIQUE INDEX uq_homeassistant_update_under_way ON homeassistant_updates ((true)) "
"WHERE state IN ('backing_up', 'installing')")
def downgrade() -> None:
op.drop_table("homeassistant_updates")
op.drop_table("homeassistant_connection")
@@ -0,0 +1,42 @@
"""Home Assistant: Steward holds a sign-in, in place of a token and a webhook
Revision ID: homeassistant_002_sign_in
Revises: homeassistant_001_connection
Create Date: 2026-10-31
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "homeassistant_002_sign_in"
down_revision: Union[str, None] = "homeassistant_001_connection"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# A saved token can't be refreshed like a sign-in, so that connection signs in again.
op.execute("DELETE FROM homeassistant_connection")
op.drop_column("homeassistant_connection", "webhook_id")
op.drop_column("homeassistant_connection", "automation")
op.drop_column("homeassistant_updates", "triggered_before")
op.alter_column("homeassistant_connection", "token", new_column_name="refresh_token")
op.add_column("homeassistant_connection", sa.Column("client_id", sa.String(500), nullable=False))
op.add_column("homeassistant_connection",
sa.Column("can_install", sa.Boolean, nullable=False, server_default=sa.false()))
op.add_column("homeassistant_connection",
sa.Column("signed_in_by", sa.String(100), nullable=False, server_default=""))
op.add_column("homeassistant_connection", sa.Column("signed_in_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.execute("DELETE FROM homeassistant_connection")
op.drop_column("homeassistant_connection", "signed_in_at")
op.drop_column("homeassistant_connection", "signed_in_by")
op.drop_column("homeassistant_connection", "can_install")
op.drop_column("homeassistant_connection", "client_id")
op.alter_column("homeassistant_connection", "refresh_token", new_column_name="token")
op.add_column("homeassistant_updates", sa.Column("triggered_before", sa.String(64), nullable=True))
op.add_column("homeassistant_connection", sa.Column("automation", sa.String(20), nullable=True))
op.add_column("homeassistant_connection", sa.Column("webhook_id", sa.Text, nullable=False))
@@ -0,0 +1,37 @@
"""Home Assistant: what the poll reads of the system, and a weekly slot for updates
Revision ID: homeassistant_003_health
Revises: homeassistant_002_sign_in
Create Date: 2026-11-01
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "homeassistant_003_health"
down_revision: Union[str, None] = "homeassistant_002_sign_in"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("homeassistant_connection",
sa.Column("health", sa.JSON, nullable=False, server_default="{}"))
op.create_table(
"homeassistant_schedule",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("enabled", sa.Boolean, nullable=False),
sa.Column("weekday", sa.Integer, nullable=False),
sa.Column("at_time", sa.String(5), nullable=False),
sa.Column("time_zone", sa.String(64), nullable=False),
sa.Column("window_minutes", sa.Integer, nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("last_slot_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("stopped", sa.Text, nullable=True),
)
def downgrade() -> None:
op.drop_table("homeassistant_schedule")
op.drop_column("homeassistant_connection", "health")
@@ -0,0 +1,34 @@
"""Home Assistant: an update the schedule installs can reboot the machine and follow it
Revision ID: homeassistant_004_reboot
Revises: homeassistant_003_health
Create Date: 2026-11-02
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "homeassistant_004_reboot"
down_revision: Union[str, None] = "homeassistant_003_health"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _under_way(states: str) -> None:
op.execute("DROP INDEX uq_homeassistant_update_under_way")
op.execute("CREATE UNIQUE INDEX uq_homeassistant_update_under_way ON homeassistant_updates ((true)) "
f"WHERE state IN ({states})")
def upgrade() -> None:
op.add_column("homeassistant_updates",
sa.Column("reboot_after", sa.Boolean, nullable=False, server_default=sa.false()))
op.add_column("homeassistant_updates", sa.Column("reboot_sent_at", sa.DateTime(timezone=True), nullable=True))
_under_way("'backing_up', 'installing', 'rebooting'")
def downgrade() -> None:
_under_way("'backing_up', 'installing'")
op.drop_column("homeassistant_updates", "reboot_sent_at")
op.drop_column("homeassistant_updates", "reboot_after")
+92
View File
@@ -0,0 +1,92 @@
# plugins/homeassistant/models.py
from __future__ import annotations
from datetime import datetime, timezone
from sqlalchemy import JSON, Boolean, DateTime, Index, Integer, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column
from steward.core.ids import time_ordered_id
from steward.models.base import Base
class HomeAssistantConnection(Base):
"""The Home Assistant instance Steward looks after; at most one row.
`refresh_token` is what Home Assistant's sign-in gave Steward, encrypted. It
has the same access as the user who signed in, whatever that is: Home
Assistant has no narrower permission. `can_install` is whether that user was
an administrator, which backups need. The last_* columns and `updates` and
`backup` and `health` are the newest poll's outcome, which the pages read. `refused` stops
all polling until someone signs in again or tries again: Home Assistant bans
an address after a few refused requests.
"""
__tablename__ = "homeassistant_connection"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
url: Mapped[str] = mapped_column(String(500), nullable=False)
client_id: Mapped[str] = mapped_column(String(500), nullable=False)
refresh_token: Mapped[str] = mapped_column(Text, nullable=False)
can_install: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
signed_in_by: Mapped[str] = mapped_column(String(100), nullable=False, default="")
signed_in_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
verify_tls: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
refused: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
last_polled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_ok_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
updates: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
backup: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
health: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
class HomeAssistantSchedule(Base):
"""The weekly slot for installing what's waiting; at most one row, none until an
admin sets it. `last_slot_at` is the slot that last fired, and `stopped` says why
that slot ended early. Times are read through steward.core.weekly."""
__tablename__ = "homeassistant_schedule"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
weekday: Mapped[int] = mapped_column(Integer, nullable=False)
at_time: Mapped[str] = mapped_column(String(5), nullable=False)
time_zone: Mapped[str] = mapped_column(String(64), nullable=False)
window_minutes: Mapped[int] = mapped_column(Integer, nullable=False)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
last_slot_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
stopped: Mapped[str | None] = mapped_column(Text, nullable=True)
class HomeAssistantUpdate(Base):
"""One update Steward was asked to install, from the click to its end. It
moves through backing_up and installing to done or failed, and its row
carries everything needed to carry on after Steward restarts. The *_before
columns are Home Assistant's own readings at the start, so "a newer backup"
is judged on its clock alone. One can be under way at a time. `reboot_after`
sends it through rebooting too, for an OS update the schedule installs.
"""
__tablename__ = "homeassistant_updates"
__table_args__ = (
Index("uq_homeassistant_update_under_way", text("(true)"), unique=True,
postgresql_where=text("state IN ('backing_up', 'installing', 'rebooting')")),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
entity_id: Mapped[str] = mapped_column(String(255), nullable=False)
title: Mapped[str] = mapped_column(String(255), nullable=False)
kind: Mapped[str] = mapped_column(String(10), nullable=False)
from_version: Mapped[str | None] = mapped_column(String(100), nullable=True)
to_version: Mapped[str | None] = mapped_column(String(100), nullable=True)
state: Mapped[str] = mapped_column(String(20), nullable=False, default="backing_up")
message: Mapped[str | None] = mapped_column(Text, nullable=True)
started_by: Mapped[str] = mapped_column(String(100), nullable=False, default="")
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
backup_before: Mapped[str | None] = mapped_column(String(64), nullable=True)
backup_event_before: Mapped[str | None] = mapped_column(String(64), nullable=True)
backup_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
install_sent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
reboot_after: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
reboot_sent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+17
View File
@@ -0,0 +1,17 @@
# plugins/homeassistant/plugin.yaml
name: homeassistant
version: "1.0.0"
description: "Keeps Home Assistant running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates after a confirmed backup, on a click or a weekly schedule."
author: "Steward"
license: "MIT"
min_app_version: "0.1.0"
repository_url: "https://git.fabledsword.com/bvandeusen/FabledSteward"
homepage: "https://git.fabledsword.com/bvandeusen/FabledSteward/src/branch/dev/plugins/homeassistant"
tags:
- home-assistant
- updates
- backups
# The connection is added and edited on the Home Assistant page
# (/plugins/homeassistant/).
config: {}
+249
View File
@@ -0,0 +1,249 @@
"""The Home Assistant page. Every GET reads Steward's own records, which the poll
fills. Only signing in, removing the connection and Install, an admin's explicit
actions, ask Home Assistant anything (System 178's ruling)."""
from __future__ import annotations
import hmac
import logging
import secrets
from datetime import datetime, timezone
from types import SimpleNamespace
from quart import Blueprint, current_app, redirect, render_template, request, session
from sqlalchemy import select
from steward.auth.middleware import require_role
from steward.core import weekly
from steward.core.audit import log_audit
from steward.core.crypto import encrypt_secret
from steward.core.formatting import zone
from steward.core.settings import public_base_url
from steward.models.users import User, UserRole
from . import client, schedule
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
from .updates import UNDER_WAY, CannotStart, start_update
logger = logging.getLogger(__name__)
homeassistant_bp = Blueprint("homeassistant", __name__, template_folder="templates")
RECENT_UPDATES = 10
_SIGN_IN = "homeassistant_sign_in" # in the session between leaving for Home Assistant and coming back
def _actor() -> tuple[str | None, str]:
return session.get("user_id"), session.get("username", "")
async def _connection(db) -> HomeAssistantConnection | None:
return (await db.execute(select(HomeAssistantConnection))).scalars().first()
def connection_says(connection, now: datetime) -> tuple[str, str]:
"""(badge state, sentence) for whether Home Assistant is answering."""
if connection.refused:
return "down", f"{connection.last_error} Steward has stopped asking until you sign in again or try again."
if connection.last_polled_at is None:
return "pending", "Not read yet: the first read runs within seconds."
if connection.last_error:
return "down", connection.last_error
return "up", "Answering."
async def _render_index(error: str | None = None, status: int = 200):
async with current_app.db_sessionmaker() as db:
connection = await _connection(db)
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
me = await db.get(User, session.get("user_id")) if session.get("user_id") else None
if connection is None:
return await render_template("homeassistant/index.html", connection=None, error=error), status
default_zone = (me.time_zone if me and me.time_zone else "UTC")
now = datetime.now(timezone.utc)
reach, says = connection_says(connection, now)
updates = connection.updates or []
backup = connection.backup or {}
tone, backup_says = backup_verdict(backup, now)
return await render_template(
"homeassistant/index.html", connection=connection, error=error, reach=reach, says=says,
installable=installable(connection),
others=[u for u in updates if u["waiting"] and not u["kind"]],
current=[u for u in updates if not u["waiting"] and u["kind"]],
runs=runs, under_way=next((r for r in runs if r.state in UNDER_WAY), None),
backup=backup, backup_tone=tone, backup_says=backup_says,
backup_at=parse_moment(backup.get("last_ok")), backup_next=parse_moment(backup.get("next")),
backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS,
health=connection.health or {}, problems=problems_open(connection.health),
plan=plan, plan_when=weekly.describe(plan) if plan else None,
plan_next=weekly.next_slot(plan, now) if plan and plan.enabled else None,
weekdays=weekly.WEEKDAYS, default_zone=default_zone), status
@homeassistant_bp.get("/")
@require_role(UserRole.viewer)
async def index():
return await _render_index()
@homeassistant_bp.post("/install")
@require_role(UserRole.admin)
async def install():
form = await request.form
entity_id = (form.get("entity_id") or "").strip()
user_id, username = _actor()
try:
run = await start_update(current_app, entity_id, username)
except CannotStart as exc:
return await _render_index(f"Nothing was started: {exc}.", 409)
await log_audit(current_app, user_id, username, "homeassistant.update_started",
entity_type="homeassistant_update", entity_id=run.id,
detail={"entity_id": entity_id, "title": run.title, "to": run.to_version})
return redirect("/plugins/homeassistant/")
@homeassistant_bp.post("/schedule")
@require_role(UserRole.admin)
async def save_schedule():
"""Set the weekly slot for installing what's waiting, or turn it off."""
form = await request.form
time_zone = (form.get("time_zone") or "UTC").strip()
try:
weekday, at_time = int(form.get("weekday", "")), (form.get("at_time") or "").strip()
weekly.parse_at(at_time)
except ValueError:
return await _render_index("Choose a day and a time.", 422)
if not 0 <= weekday <= 6 or (zone(time_zone) is timezone.utc and time_zone.upper() != "UTC"):
return await _render_index(f"Choose a day, and a known time zone rather than “{time_zone}”.", 422)
row = await schedule.save(current_app, enabled=bool(form.get("enabled")), weekday=weekday,
at_time=at_time, time_zone=time_zone)
await log_audit(current_app, *_actor(), "homeassistant.schedule_saved", entity_type="homeassistant_schedule",
entity_id=row.id, detail={"enabled": row.enabled, "when": weekly.describe(row)})
return redirect("/plugins/homeassistant/")
# ── The connection ───────────────────────────────────────────────────────────
async def _render_form(connection, url: str | None = None, verify_tls: bool | None = None,
error: str | None = None, status: int = 200):
return await render_template(
"homeassistant/form.html", connection=connection, error=error,
url=url if url is not None else (connection.url if connection else ""),
verify_tls=verify_tls if verify_tls is not None else (connection.verify_tls if connection else True)), status
@homeassistant_bp.get("/connection")
@require_role(UserRole.admin)
async def connection_form():
async with current_app.db_sessionmaker() as db:
connection = await _connection(db)
return await _render_form(connection)
@homeassistant_bp.post("/connection/sign-in")
@require_role(UserRole.admin)
async def sign_in():
"""Send the browser to Home Assistant's own sign-in page."""
form = await request.form
url = (form.get("url") or "").strip().rstrip("/")
verify_tls = bool(form.get("verify_tls"))
if not url.startswith(("http://", "https://")):
async with current_app.db_sessionmaker() as db:
connection = await _connection(db)
return await _render_form(connection, url, verify_tls, "Enter Home Assistant's address: http://… or https://…", 422)
base, state = public_base_url(request), secrets.token_urlsafe(32)
client_id = f"{base}/"
session[_SIGN_IN] = {"state": state, "url": url, "verify_tls": verify_tls, "client_id": client_id}
return redirect(client.sign_in_url(url, client_id, f"{base}/plugins/homeassistant/connection/callback", state))
@homeassistant_bp.get("/connection/callback")
@require_role(UserRole.admin)
async def sign_in_callback():
"""Where Home Assistant sends the browser back. A GET that asks Home Assistant
something, because the admin's sign-in has only just happened and its code
lasts minutes. The state shows the sign-in began here, for this admin."""
pending = session.pop(_SIGN_IN, None)
state, code = request.args.get("state", ""), request.args.get("code", "")
async with current_app.db_sessionmaker() as db:
connection = await _connection(db)
if not pending or not hmac.compare_digest(pending["state"].encode(), state.encode()):
return await _render_form(connection, error="That sign-in didn't start from this page, so it wasn't used. "
"If it keeps happening, check Public base URL in Settings → General: "
"Steward must be opened at that address.", status=400)
url, verify_tls, client_id = pending["url"], pending["verify_tls"], pending["client_id"]
if not code:
return await _render_form(connection, url, verify_tls, "Home Assistant didn't finish the sign-in. Nothing changed.", 400)
try:
refresh_token, token = await client.redeem_code(url, client_id, code, verify_tls)
can_install = await client.is_admin(url, token, verify_tls)
except (client.Refused, client.Unreachable) as exc:
return await _render_form(connection, url, verify_tls, f"The sign-in couldn't be finished: {exc}", 502)
user_id, username = _actor()
previous = None
async with current_app.db_sessionmaker() as db:
async with db.begin():
connection = await _connection(db)
if connection is None:
connection = HomeAssistantConnection(url=url, client_id=client_id, refresh_token="")
db.add(connection)
else:
previous = (connection.url, connection.client_id, connection.refresh_token, connection.verify_tls)
connection.url, connection.client_id, connection.verify_tls = url, client_id, verify_tls
connection.refresh_token = encrypt_secret(refresh_token)
connection.can_install, connection.signed_in_by = can_install, username
connection.signed_in_at = datetime.now(timezone.utc)
connection.refused, connection.last_error, connection.last_polled_at = False, None, None
await db.flush()
connection_id = connection.id
if previous is not None:
await _end_sign_in(SimpleNamespace(url=previous[0], client_id=previous[1], refresh_token=previous[2],
verify_tls=previous[3]))
await log_audit(current_app, user_id, username, "homeassistant.connection_saved",
entity_type="homeassistant_connection", entity_id=connection_id,
detail={"name": url, "can_install": can_install})
return redirect("/plugins/homeassistant/")
async def _end_sign_in(connection) -> bool:
"""End a sign-in Steward no longer needs, so it doesn't stay valid on Home
Assistant. It can be ended there by hand too, so failing here stops nothing."""
try:
await client.revoke(connection)
except (client.Refused, client.Unreachable) as exc:
logger.warning("Home Assistant: couldn't end Steward's old sign-in: %s", exc)
return False
return True
@homeassistant_bp.post("/connection/retry")
@require_role(UserRole.admin)
async def retry_connection():
"""Ask again after a refusal, once a ban has been dealt with."""
async with current_app.db_sessionmaker() as db:
async with db.begin():
connection = await _connection(db)
if connection is not None:
connection.refused, connection.last_polled_at = False, None
return redirect("/plugins/homeassistant/")
@homeassistant_bp.post("/connection/delete")
@require_role(UserRole.admin)
async def delete_connection():
async with current_app.db_sessionmaker() as db:
async with db.begin():
connection = await _connection(db)
if connection is None:
return redirect("/plugins/homeassistant/")
connection_id, url = connection.id, connection.url
await db.delete(connection)
ended = await _end_sign_in(connection)
await log_audit(current_app, *_actor(), "homeassistant.connection_removed",
entity_type="homeassistant_connection", entity_id=connection_id, detail={"name": url, "sign_in_ended": ended})
if not ended:
return await _render_index("Removed from Steward, but Home Assistant couldn't be reached to end the sign-in. "
"End it yourself under Security on that user's profile in Home Assistant.")
return redirect("/plugins/homeassistant/")
+95
View File
@@ -0,0 +1,95 @@
# plugins/homeassistant/schedule.py
"""Installing what's waiting at a weekly slot: one at a time through the same path
as the page's Install, each after its own confirmed backup. Add-ons go first and
the OS last, as it reboots the machine. The first failure ends the slot, leaving the
rest for someone to look at. Everything is on the rows, so a restart of Steward or
of Home Assistant only delays the next step."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from steward.core import weekly
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
from .service import installable
from .updates import CannotStart, start_update, under_way
ACTOR = "schedule"
WINDOW_MINUTES = 240
_ORDER = {"addon": 0, "core": 1, "os": 2}
def next_to_install(waiting: list[dict], tried: set[str]) -> dict | None:
left = [u for u in waiting if u["entity_id"] not in tried and not u["in_progress"]]
return min(left, key=lambda u: _ORDER[u["kind"]], default=None)
async def _stop(app, schedule_id: str, why: str) -> None:
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantSchedule, schedule_id)
if row is not None:
row.stopped = why
async def _fire(app, schedule_id: str, slot: datetime) -> None:
"""Open the slot, and have the next poll say what's waiting at this moment."""
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantSchedule, schedule_id)
row.last_slot_at, row.stopped = slot, None
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if connection is not None:
connection.last_polled_at = None
async def step(app, transport=None) -> None:
"""Fire a slot that has come, or start the next update of one that's open."""
now = datetime.now(timezone.utc)
async with app.db_sessionmaker() as session:
schedule = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if schedule is None or connection is None:
return
slot = weekly.is_due(schedule, now)
if slot is not None:
await _fire(app, schedule.id, slot)
return
slot = weekly.fired_slot(schedule)
if not schedule.enabled or schedule.stopped or slot is None \
or now - slot >= timedelta(minutes=schedule.window_minutes):
return
# Only on a fresh, good read: Home Assistant may be restarting from the last update.
if connection.refused or connection.last_polled_at is None or connection.last_error:
return
if await under_way(session) is not None:
return
runs = (await session.execute(select(HomeAssistantUpdate).where(
HomeAssistantUpdate.started_at >= slot))).scalars().all()
failed = next((r for r in runs if r.state == "failed"), None)
if failed is not None:
await _stop(app, schedule.id, f"{failed.title}: {failed.message} Nothing more was installed in this slot.")
return
update = next_to_install(installable(connection), {r.entity_id for r in runs})
if update is None:
return
try:
await start_update(app, update["entity_id"], ACTOR, transport, reboot_after=update["kind"] == "os")
except CannotStart as exc:
await _stop(app, schedule.id, f"{update['title']} wasn't started: {exc}. Nothing more was installed in this slot.")
async def save(app, *, enabled: bool, weekday: int, at_time: str, time_zone: str) -> HomeAssistantSchedule:
"""Set the slot. Its first is the next one from now, not one already past."""
async with app.db_sessionmaker() as session:
async with session.begin():
row = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
if row is None:
row = HomeAssistantSchedule(window_minutes=WINDOW_MINUTES)
session.add(row)
row.enabled, row.weekday, row.time_zone = enabled, weekday, time_zone
row.at_time = f"{weekly.parse_at(at_time):%H:%M}"
row.last_slot_at, row.stopped = datetime.now(timezone.utc), None
return row
+116
View File
@@ -0,0 +1,116 @@
# plugins/homeassistant/scheduler.py
from __future__ import annotations
import asyncio
import logging
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from sqlalchemy import select
from steward.core.scheduler import ScheduledTask
from . import client, schedule
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_health, read_snapshot
from .updates import advance
if TYPE_CHECKING:
from quart import Quart
logger = logging.getLogger(__name__)
# An update under way is followed each tick, with one small read of its entity.
TICK_SECONDS = 15
# The whole entity list is a few hundred KB, and updates and backups change rarely.
POLL_SECONDS = 300
# A step past this is given up until the next tick, whatever the client's own
# timeouts did: a slow drip of bytes can keep each read inside its timeout for ever.
STEP_DEADLINE_SECONDS = 60
def make_tick_task(app: "Quart") -> ScheduledTask:
busy = asyncio.Lock()
async def tick() -> None:
if busy.locked():
return
async with busy:
for step in (poll, advance, schedule.step):
try:
await asyncio.wait_for(step(app), STEP_DEADLINE_SECONDS)
except TimeoutError:
logger.warning("Home Assistant %s gave up after %d s", step.__name__, STEP_DEADLINE_SECONDS)
return ScheduledTask(name="homeassistant_tick", coro_factory=tick,
interval_seconds=TICK_SECONDS, run_on_startup=True)
def is_due(connection, now: datetime) -> bool:
"""Whether to poll. Never after a refusal: another refused request counts
toward Home Assistant banning this address."""
if connection.refused:
return False
if connection.last_polled_at is None:
return True
return (now - connection.last_polled_at).total_seconds() >= POLL_SECONDS - TICK_SECONDS / 2
async def update_failed(session) -> bool:
"""Whether the newest update failed, or the schedule couldn't start one."""
stopped = (await session.execute(select(HomeAssistantSchedule.stopped))).scalars().first()
newest = (await session.execute(select(HomeAssistantUpdate.state).order_by(
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
return bool(stopped) or newest == "failed"
async def poll(app: "Quart", transport=None, connect=None) -> None:
from steward.core.alerts import MetricBatch
now = datetime.now(timezone.utc)
async with app.db_sessionmaker() as session:
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if connection is None or not is_due(connection, now):
return
snapshot, health, error, refused = None, None, None, False
try:
token = await client.access_token(connection, transport)
snapshot = read_snapshot(await client.read_states(connection.url, token, connection.verify_tls, transport))
try:
health = read_health(await client.read_system(connection.url, token, connection.verify_tls, connect))
except client.Unreachable as exc:
# Updates and backups were read, so it's answering. The last report stands, and says it's old.
health = {**(connection.health or {}), "error": str(exc)}
except client.Refused as exc:
snapshot, error, refused = None, str(exc), True
except client.Unreachable as exc:
error = str(exc)
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantConnection, connection.id)
if row is None: # removed while it was being polled
return
batch = MetricBatch()
row.last_polled_at, row.last_error, row.refused = now, error, refused
if snapshot is None:
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=0.0, recorded_at=now)
else:
row.last_ok_at = now
row.updates, row.backup, row.health = snapshot["updates"], snapshot["backup"], health
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=1.0, recorded_at=now)
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="updates_waiting",
value=len(installable(row)), recorded_at=now)
age = backup_age_hours(snapshot["backup"], now)
if age is not None:
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="backup_age_hours",
value=age, recorded_at=now)
if not health.get("error"):
if health["repairs_read"]:
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="problems_open",
value=len(health["problems"]), recorded_at=now)
if health["disk_free_pct"] is not None:
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="disk_free_pct",
value=health["disk_free_pct"], recorded_at=now)
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="update_failed",
value=1.0 if await update_failed(session) else 0.0, recorded_at=now)
await batch.record(session)
+161
View File
@@ -0,0 +1,161 @@
# plugins/homeassistant/service.py
"""Reading what Home Assistant reports: which updates it knows of, which of them
Steward may install, how its backups stand, and whether the system has a problem."""
from __future__ import annotations
from datetime import datetime, timezone
SOURCE = "homeassistant"
RESOURCE = "homeassistant"
# Backups run daily, so past a day and a bit one has been missed.
STALE_BACKUP_HOURS = 26
LOW_DISK_PCT = 10
CORE = "update.home_assistant_core_update"
OS = "update.home_assistant_operating_system_update"
SUPERVISOR = "update.home_assistant_supervisor_update"
LAST_BACKUP = "sensor.backup_last_successful_automatic_backup"
NEXT_BACKUP = "sensor.backup_next_scheduled_automatic_backup"
BACKUP_MANAGER = "sensor.backup_backup_manager_state"
BACKUP_EVENT = "event.backup_automatic_backup"
_KIND_WORDS = {"core": "Core", "os": "Operating system", "addon": "Add-on"}
_KIND_ORDER = {"core": 0, "os": 1, "addon": 2, None: 3}
def classify(entity: dict) -> str | None:
""""core", "os" or "addon" for an update Steward may install; None for the
rest: Supervisor updates itself, and firmware and custom integrations are
Home Assistant's own to offer."""
entity_id = entity.get("entity_id")
if entity_id == CORE:
return "core"
if entity_id == OS:
return "os"
if entity_id == SUPERVISOR:
return None
picture = (entity.get("attributes") or {}).get("entity_picture") or ""
return "addon" if picture.startswith("/api/hassio/addons/") else None
def _update(entity: dict) -> dict:
a = entity.get("attributes") or {}
kind = classify(entity)
url = a.get("release_url") or ""
return {"entity_id": entity["entity_id"], "kind": kind, "kind_word": _KIND_WORDS.get(kind, "Other"),
"title": a.get("title") or a.get("friendly_name") or entity["entity_id"],
"installed": a.get("installed_version"), "latest": a.get("latest_version"),
"waiting": entity.get("state") == "on", "in_progress": bool(a.get("in_progress")),
"auto_update": bool(a.get("auto_update")),
"release_url": url if url.startswith(("http://", "https://")) else None}
def _state(entity: dict | None) -> str | None:
value = (entity or {}).get("state")
return None if value in (None, "unknown", "unavailable") else value
def parse_moment(value: str | None) -> datetime | None:
try:
moment = datetime.fromisoformat(value) if value else None
except ValueError:
return None
if moment is not None and moment.tzinfo is None:
moment = moment.replace(tzinfo=timezone.utc)
return moment
def is_newer(value: str | None, than: str | None) -> bool:
"""Whether reading `value` is a later moment than `than`, where `than` may be never."""
moment, before = parse_moment(value), parse_moment(than)
return moment is not None and (before is None or moment > before)
def read_snapshot(states: list[dict]) -> dict:
"""What a poll keeps of Home Assistant's entities."""
by_id = {e.get("entity_id"): e for e in states}
updates = [_update(e) for i, e in by_id.items() if (i or "").startswith("update.") and e.get("state") != "unavailable"]
updates.sort(key=lambda u: (not u["waiting"], _KIND_ORDER[u["kind"]], u["title"].lower()))
event = by_id.get(BACKUP_EVENT) or {}
event_attrs = event.get("attributes") or {}
return {
"updates": updates,
"backup": {"last_ok": _state(by_id.get(LAST_BACKUP)), "next": _state(by_id.get(NEXT_BACKUP)),
"manager": _state(by_id.get(BACKUP_MANAGER)), "last_result": event_attrs.get("event_type"),
"failed_reason": event_attrs.get("failed_reason"), "known": LAST_BACKUP in by_id},
}
def backup_age_hours(backup: dict | None, now: datetime) -> float | None:
last = parse_moment((backup or {}).get("last_ok"))
return None if last is None else max(0.0, (now - last).total_seconds() / 3600)
def installable(connection) -> list[dict]:
"""The waiting updates Steward may install."""
return [u for u in (connection.updates or []) if u["waiting"] and u["kind"]]
def backup_verdict(backup: dict | None, now: datetime) -> tuple[str, str]:
"""(tone, sentence) for how the backups stand."""
backup = backup or {}
if not backup.get("known"):
return "warn", "Home Assistant reports no automatic backups. Turn them on under Settings → System → Backups."
age = backup_age_hours(backup, now)
if age is None:
return "warn", "Home Assistant has no successful automatic backup yet."
if backup.get("last_result") == "failed":
reason = backup.get("failed_reason")
return "warn", f"The newest backup attempt failed{f' ({reason})' if reason else ''}."
if age > STALE_BACKUP_HOURS:
return "warn", f"The last good backup is {age:.0f} hours old, past the {STALE_BACKUP_HOURS} hours expected."
return "ok", "Backups are up to date."
def _issue_title(issue: dict, titles: dict) -> str:
key = issue.get("translation_key") or issue.get("issue_id") or ""
title = titles.get(f"component.{issue.get('domain')}.issues.{key}.title")
if not title:
return key.removeprefix("issue_").replace("_", " ").capitalize() or "A repair is open"
for name, value in (issue.get("translation_placeholders") or {}).items():
title = title.replace("{" + name + "}", str(value))
return title
def read_health(found: dict) -> dict:
"""What a poll keeps of client.read_system: the problems to show, and the disk.
A part Home Assistant wouldn't answer is left out rather than read as fine:
`repairs_read` says whether the repairs were seen, and `supervised` whether a
Supervisor answered, as it doesn't on every install or for every user."""
problems = []
if (found.get("config") or {}).get("recovery_mode"):
problems.append({"kind": "recovery", "severity": "critical",
"title": "Home Assistant started in recovery mode: its configuration didn't load."})
issues = found.get("issues")
for issue in issues or []:
if not issue.get("ignored") and not issue.get("dismissed_version"):
problems.append({"kind": "repair", "severity": issue.get("severity") or "warning",
"title": _issue_title(issue, found.get("titles") or {})})
resolution = found.get("resolution") or {}
for word in ("unhealthy", "unsupported"):
for reason in resolution.get(word) or []:
problems.append({"kind": "supervisor", "severity": "error" if word == "unhealthy" else "warning",
"title": f"The Supervisor calls this system {word}: {str(reason).replace('_', ' ')}."})
host = found.get("host") or {}
free, used, total = host.get("disk_free"), host.get("disk_used"), host.get("disk_total")
free_pct = round(100 * free / total, 1) if free is not None and total else None
# Used and free don't add up to the total: the system reserves some of the disk.
used_pct = round(100 * used / total, 1) if used is not None and total else None
if free_pct is not None and free_pct < LOW_DISK_PCT:
problems.append({"kind": "disk", "severity": "error", "title": f"Only {free:g} GB of {total:g} GB free."})
boots = found.get("boots") or {}
for addon in found.get("addons") or []:
if boots.get(addon.get("slug")) == "auto":
problems.append({"kind": "addon", "severity": "error",
"title": f"{addon.get('name') or addon.get('slug')} should start on its own and isn't running."})
return {"repairs_read": issues is not None, "supervised": found.get("resolution") is not None, "problems": problems,
"disk_free_gb": free, "disk_used_gb": used, "disk_total_gb": total,
"disk_free_pct": free_pct, "disk_used_pct": used_pct}
def problems_open(health: dict | None) -> list[dict]:
return (health or {}).get("problems") or []
@@ -0,0 +1,50 @@
{# plugins/homeassistant/templates/homeassistant/form.html — where Home Assistant
is, what signing in gives Steward, and leaving for its sign-in page. #}
{% extends "base.html" %}
{% from "_macros.html" import crumbs, page_header %}
{% block title %}Home Assistant connection — Steward{% endblock %}
{% block breadcrumb %}{{ crumbs([("Home Assistant", "/plugins/homeassistant/"), ("Connection", "")]) }}{% endblock %}
{% block content %}
{{ page_header("Home Assistant connection") }}
<div class="form-page">
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
{% if connection %}
<p class="field-hint">Signed in{% if connection.signed_in_at %} {{ connection.signed_in_at|ago }}{% endif %}{% if connection.signed_in_by %} by {{ connection.signed_in_by }}{% endif %}, as {{ "an administrator: Steward can install updates" if connection.can_install else "a user who isn't an administrator: Steward shows status only" }}.</p>
{% endif %}
<form method="post" action="/plugins/homeassistant/connection/sign-in">{{ csrf_field() }}
<section class="card form-section" aria-labelledby="ha-sign-in">
<h2 id="ha-sign-in" class="section-title">Sign in with Home Assistant</h2>
<div class="field">
<label class="field-label" for="ha-url">Address</label>
<input class="input mono" id="ha-url" name="url" required maxlength="500" value="{{ url or '' }}" placeholder="http://homeassistant.local:8123" autocomplete="off">
<span class="field-hint">Both your browser and Steward need to reach it at this address.</span>
</div>
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.9rem;">
<input type="checkbox" name="verify_tls" value="on"{% if verify_tls %} checked{% endif %}> Check its HTTPS certificate
</label>
<p class="tone-warn"><strong>Steward stores a credential with the same access as the user who signs in.</strong> Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates.</p>
<p class="field-hint"><strong>We recommend a service account:</strong> a Home Assistant user made just for Steward (Settings → People → Add person). What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.</p>
<p class="field-hint" style="margin-bottom:0;">Installing updates needs an administrator. With any other user, Steward shows status only.</p>
</section>
<div class="form-actions">
<button type="submit" class="btn">{{ "Sign in again" if connection else "Continue to Home Assistant" }}</button>
<a href="/plugins/homeassistant/" class="btn btn-ghost">Cancel</a>
</div>
</form>
{% if connection %}
<section class="card danger-zone" aria-labelledby="ha-remove">
<div>
<h2 id="ha-remove" class="section-title">Remove this connection</h2>
<p class="field-hint">Steward stops reading Home Assistant, forgets its credential and ends the sign-in there.</p>
</div>
<form method="post" action="/plugins/homeassistant/connection/delete"
data-confirm="Remove the Home Assistant connection? Steward stops reading it and forgets its credential.">{{ csrf_field() }}
<button class="btn btn-danger" type="submit">Remove connection</button>
</form>
</section>
{% endif %}
</div>
{% endblock %}
@@ -0,0 +1,150 @@
{# plugins/homeassistant/templates/homeassistant/index.html — whether Home
Assistant answers, which updates are waiting with Install beside each, how its
backups stand, and the updates Steward has run. Refreshes itself while one is
under way. #}
{% extends "base.html" %}
{% from "_macros.html" import page_header, status_badge, stat_tile, empty_state %}
{% block title %}Home Assistant — Steward{% endblock %}
{% block content %}
{% set admin = session.user_role == 'admin' %}
{% call page_header("Home Assistant", connection.url if connection else none) %}
{% if admin and connection %}<a class="btn btn-sm btn-ghost" href="/plugins/homeassistant/connection">Connection</a>{% endif %}
{% endcall %}
{% if not connection %}
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
<div class="card">{{ empty_state("Not connected. Sign in to a Home Assistant instance to see its waiting updates and backups, and to install updates after a confirmed backup.",
"Connect" if admin else none, "/plugins/homeassistant/connection") }}</div>
{% else %}
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
<div id="ha-body"{% if under_way %} hx-get="/plugins/homeassistant/" hx-select="#ha-body" hx-trigger="every 15s" hx-swap="outerHTML"{% endif %}>
<p class="page-verdict">
{% if under_way %}{{ status_badge('pending', 'Updating') }}
{{ under_way.title }}: {{ {'backing_up': "taking a backup first", 'rebooting': "rebooting into " ~ under_way.to_version}.get(under_way.state, "installing " ~ under_way.to_version) }}. Home Assistant may stop answering for a while.
{% elif reach == 'up' %}{{ status_badge('up', 'Answering') }}
{% if installable %}{{ installable|length }} update{{ "s" if installable|length != 1 }} waiting.{% else %}Everything Steward can update is up to date.{% endif %}
Read {{ connection.last_polled_at|ago }}.
{% elif reach == 'down' %}{{ status_badge('down', 'Not answering') }} <span class="tone-crit">{{ says }}</span>
{% if connection.last_ok_at %}Last answered {{ connection.last_ok_at|ago }}; what's below is from then.{% endif %}
{% if admin and connection.refused %}<form method="post" action="/plugins/homeassistant/connection/retry" style="display:inline;">{{ csrf_field() }}<button class="btn btn-sm btn-ghost" type="submit">Try again</button></form>{% endif %}
{% else %}{{ status_badge('pending', 'Not read yet') }} {{ says }}{% endif %}
</p>
{% if not connection.can_install %}
<p class="tone-warn">The Home Assistant user who signed in isn't an administrator, so updates can't be installed from here.
{% if admin %}<a href="/plugins/homeassistant/connection">Sign in again</a> as an administrator to install them.{% endif %}</p>
{% endif %}
<div class="stat-strip">
{{ stat_tile("System problems", problems|length if health.repairs_read else "—", state="degraded" if problems else none) }}
{{ stat_tile("Storage", ("%s of %s GB (%.0f%%)"|format(health.disk_used_gb, health.disk_total_gb, health.disk_used_pct)) if health.disk_used_pct is number else "—") }}
{{ stat_tile("Updates waiting", installable|length if connection.last_ok_at else "—") }}
{{ stat_tile("Last good backup", (backup_at|ago) if backup_at else "—", state="degraded" if backup_tone == 'warn' else none) }}
{{ stat_tile("Next backup", (backup_next|ago) if backup_next else "—") }}
</div>
<p class="{{ 'tone-warn' if backup_tone == 'warn' else 'field-hint' }}">{{ backup_says }}</p>
<section class="card" aria-labelledby="ha-system">
<h2 id="ha-system" class="section-title">System</h2>
{% if problems %}
<table class="table">
<thead><tr><th scope="col">Problem</th><th scope="col">Severity</th></tr></thead>
<tbody>
{% for p in problems %}<tr><td>{{ p.title }}</td><td>{{ status_badge('down' if p.severity in ('critical', 'error') else 'degraded', p.severity|capitalize) }}</td></tr>{% endfor %}
</tbody>
</table>
<p class="field-hint">Fix these in Home Assistant: Settings → System → Repairs.</p>
{% elif health.repairs_read %}
<p class="field-hint">No repairs open, and nothing wrong reported.</p>
{% elif connection.last_ok_at %}
<p class="tone-warn">Home Assistant wouldn't share its repairs with this user. {% if admin %}<a href="/plugins/homeassistant/connection">Sign in again</a> as an administrator to see them.{% endif %}</p>
{% endif %}
{% if health.repairs_read and not health.supervised %}<p class="field-hint">No Supervisor answered, so disk space and add-ons aren't checked here.</p>{% endif %}
{% if health.error %}<p class="tone-warn" style="margin-bottom:0;">This is from an earlier read. {{ health.error }}</p>{% endif %}
</section>
<section class="card" aria-labelledby="ha-updates">
<h2 id="ha-updates" class="section-title">Updates</h2>
{% if installable %}
<table class="table">
<thead><tr><th scope="col">What</th><th scope="col">Now</th><th scope="col">New</th>{% if admin %}<th scope="col"></th>{% endif %}</tr></thead>
<tbody>
{% for u in installable %}
<tr>
<td>{{ u.title }}<div class="cell-sub">{{ u.kind_word }}{% if u.auto_update %}, updates itself{% endif %}</div></td>
<td class="mono">{{ u.installed or "—" }}</td>
<td class="mono">{% if u.release_url %}<a href="{{ u.release_url }}" rel="noopener noreferrer" target="_blank">{{ u.latest or "—" }}</a>{% else %}{{ u.latest or "—" }}{% endif %}</td>
{% if admin %}<td style="text-align:right;">
{% if connection.can_install and not under_way and not connection.refused and not u.in_progress %}
<form method="post" action="/plugins/homeassistant/install"
data-confirm="Install {{ u.title }} {{ u.latest }}? Steward takes a backup first and won't install without it. {{ 'Home Assistant restarts.' if u.kind == 'core' else ('It takes effect once you reboot the machine, which Steward leaves to you.' if u.kind == 'os' else 'The add-on restarts.') }}">{{ csrf_field() }}
<input type="hidden" name="entity_id" value="{{ u.entity_id }}">
<button class="btn btn-sm" type="submit">Install</button>
</form>
{% endif %}
</td>{% endif %}
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="field-hint" style="margin:0;">Nothing waiting.</p>
{% endif %}
{% if current %}<p class="field-hint">Up to date: {% for u in current %}{{ u.title }} {{ u.installed }}{{ ", " if not loop.last }}{% endfor %}.</p>{% endif %}
{% if others %}<p class="field-hint" style="margin-bottom:0;">Waiting, for Home Assistant to install: {% for u in others %}{{ u.title }}{{ ", " if not loop.last }}{% endfor %}. Steward installs Core, OS and add-on updates only.</p>{% endif %}
</section>
<section class="card" aria-labelledby="ha-schedule">
<h2 id="ha-schedule" class="section-title">Scheduled updates</h2>
{% if plan and plan.enabled %}
<p class="field-hint">{{ plan_when }}. Next: {{ plan_next|ago }}. Add-ons first, then Core, then the OS and its reboot, each after a confirmed backup. A failure ends the run and the rest wait.</p>
{% else %}
<p class="field-hint">Off. Updates are only installed when someone chooses Install.</p>
{% endif %}
{% if plan and plan.stopped %}<p class="tone-warn">The last run stopped early. {{ plan.stopped }}</p>{% endif %}
{% if admin %}
<p class="tone-warn">A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch. An update can also change how automations behave.</p>
<form method="post" action="/plugins/homeassistant/schedule" style="display:flex;flex-wrap:wrap;gap:0.75rem;align-items:flex-end;">{{ csrf_field() }}
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.4rem;">
<input type="checkbox" name="enabled" value="on"{% if plan and plan.enabled %} checked{% endif %}> Install on a schedule
</label>
<div class="field" style="margin-bottom:0;">
<label class="field-label" for="ha-day">Day</label>
<select class="input" id="ha-day" name="weekday">{% for d in weekdays %}<option value="{{ loop.index0 }}"{% if loop.index0 == (plan.weekday if plan else 5) %} selected{% endif %}>{{ d }}</option>{% endfor %}</select>
</div>
<div class="field" style="margin-bottom:0;">
<label class="field-label" for="ha-time">Time</label>
<input class="input" id="ha-time" type="time" name="at_time" required value="{{ plan.at_time if plan else '05:00' }}">
</div>
<div class="field" style="margin-bottom:0;">
<label class="field-label" for="ha-zone">Time zone</label>
<input class="input" id="ha-zone" name="time_zone" required value="{{ plan.time_zone if plan else default_zone }}">
</div>
<button class="btn btn-sm" type="submit">Save</button>
</form>
{% endif %}
</section>
{% if runs %}
<section class="card" aria-labelledby="ha-runs">
<h2 id="ha-runs" class="section-title">Updates run from Steward</h2>
<table class="table">
<thead><tr><th scope="col">What</th><th scope="col">Versions</th><th scope="col">Outcome</th><th scope="col">Started</th></tr></thead>
<tbody>
{% for r in runs %}
<tr>
<td>{{ r.title }}<div class="cell-sub">by {{ r.started_by or "Steward" }}</div></td>
<td class="mono">{{ r.from_version or "—" }} → {{ r.to_version or "—" }}</td>
<td>{% if r.state == 'done' %}{{ status_badge('up', 'Done') }}
{% elif r.state == 'failed' %}{{ status_badge('down', 'Failed') }}<div class="cell-sub">{{ r.message }}</div>
{% else %}{{ status_badge('pending', {'backing_up': 'Taking a backup', 'rebooting': 'Rebooting'}.get(r.state, 'Installing')) }}{% endif %}</td>
<td>{{ r.started_at|ago }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endif %}
</div>
{% endif %}
{% endblock %}
+34
View File
@@ -0,0 +1,34 @@
# plugins/homeassistant/timeline.py
"""What Home Assistant maintenance puts on the timeline: each update once it ends."""
from __future__ import annotations
from steward.timeline import catalog
from steward.timeline.record import record_or_update
TYPES = (
("homeassistant.updated", "Home Assistant updated", "automation", "service", True,
"Steward installed a Home Assistant update after a confirmed backup."),
("homeassistant.update_failed", "Home Assistant update failed", "automation", "service", True,
"A Home Assistant update Steward was asked to install didn't finish."),
)
def register() -> None:
for t in TYPES:
catalog.register_type(*t)
from .models import HomeAssistantUpdate
catalog.register_ref("homeassistant_update", HomeAssistantUpdate)
async def update_entry(session, run) -> None:
done = run.state == "done"
versions = f"{run.from_version} to {run.to_version}" if run.to_version else "a new version"
summary = (f"{run.title} updated from {versions}" if done
else f"{run.title} update failed: {run.message}")
await record_or_update(
session, ("homeassistant_update", run.id),
type="homeassistant.updated" if done else "homeassistant.update_failed",
at=run.started_at, until=run.finished_at, summary=summary, subject=run.title,
detail={"entity_id": run.entity_id, "from": run.from_version, "to": run.to_version,
"url": "/plugins/homeassistant/"},
actor=run.started_by or "Steward")
+259
View File
@@ -0,0 +1,259 @@
# plugins/homeassistant/updates.py
"""Installing one update: take a backup, confirm it landed, install, and follow
it to its end. Its row holds where it has got to, so Steward restarting, or Home
Assistant restarting as an update does, only delays the next step."""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from . import client
from .models import HomeAssistantConnection, HomeAssistantUpdate
from .service import BACKUP_EVENT, LAST_BACKUP, is_newer, parse_moment
from .timeline import update_entry
logger = logging.getLogger(__name__)
UNDER_WAY = ("backing_up", "installing", "rebooting")
BACKUP_DEADLINE_S = 30 * 60
REBOOT_DEADLINE_S = 20 * 60
# An OS update downloads and writes a whole system image, which is slow on a Pi. Home
# Assistant reports the new version as soon as that's written, before the reboot that
# makes it run, and raises a repair asking for that reboot. The schedule's own OS
# updates make that reboot; one started by a person leaves it to them.
INSTALL_DEADLINE_S = {"addon": 20 * 60, "core": 30 * 60, "os": 45 * 60}
class CannotStart(Exception):
"""Why an update wasn't started, said to the person."""
async def _mark_refused(app, connection_id: str, said: str) -> None:
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantConnection, connection_id)
if row is not None:
row.refused, row.last_error = True, said
async def under_way(session) -> HomeAssistantUpdate | None:
return (await session.execute(select(HomeAssistantUpdate).where(
HomeAssistantUpdate.state.in_(UNDER_WAY)))).scalars().first()
async def start_update(app, entity_id: str, username: str, transport=None, *, reboot_after: bool = False) -> HomeAssistantUpdate:
"""Start installing `entity_id`, an explicit action that may contact Home
Assistant. Raises CannotStart, with nothing started, unless the update is
still waiting and the signed-in user may install it. `reboot_after` reboots
the machine once it's written, and it's done when the new version is running."""
async with app.db_sessionmaker() as session:
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if connection is None:
raise CannotStart("Home Assistant isn't connected")
if connection.refused:
raise CannotStart("Home Assistant refused Steward's sign-in")
if await under_way(session) is not None:
raise CannotStart("another update is under way")
if not connection.can_install:
raise CannotStart("the Home Assistant user who signed in isn't an administrator")
url, verify = connection.url, connection.verify_tls
known = next((u for u in connection.updates or [] if u["entity_id"] == entity_id), None)
if known is None or not known["kind"]:
raise CannotStart("that isn't a Core, OS or add-on update")
try:
token = await client.access_token(connection, transport)
current = await client.read_state(url, entity_id, token, verify, transport)
backup = await client.read_state(url, LAST_BACKUP, token, verify, transport)
event = await client.read_state(url, BACKUP_EVENT, token, verify, transport)
except client.Refused as exc:
await _mark_refused(app, connection.id, str(exc))
raise CannotStart(str(exc).rstrip(".")) from None
except client.Unreachable as exc:
raise CannotStart(str(exc).rstrip(".")) from None
if backup is None:
raise CannotStart("Home Assistant has no automatic backups to confirm one with")
if current is None or current.get("state") != "on":
raise CannotStart("that update is no longer waiting")
attrs = current.get("attributes") or {}
run = HomeAssistantUpdate(
entity_id=entity_id, title=known["title"], kind=known["kind"],
from_version=attrs.get("installed_version"), to_version=attrs.get("latest_version"),
started_by=username, backup_before=backup.get("state"),
backup_event_before=(event or {}).get("state"), reboot_after=reboot_after)
async with app.db_sessionmaker() as session:
try:
async with session.begin():
session.add(run)
await session.flush()
run_id = run.id
except IntegrityError:
raise CannotStart("another update is under way") from None
try:
await client.call_action(url, "backup", "create_automatic", {}, token, verify, transport)
except client.Refused as exc:
await _mark_refused(app, connection.id, str(exc))
await _finish(app, run_id, "failed", f"The backup couldn't be started: {exc} Nothing was installed.")
raise CannotStart(str(exc).rstrip(".")) from None
except (client.Unreachable, client.Rejected) as exc:
await _finish(app, run_id, "failed", f"The backup couldn't be started: {exc} Nothing was installed.")
raise CannotStart(str(exc).rstrip(".")) from None
async with app.db_sessionmaker() as session:
return await session.get(HomeAssistantUpdate, run_id)
async def _finish(app, run_id: str, state: str, message: str | None) -> None:
async with app.db_sessionmaker() as session:
async with session.begin():
run = await session.get(HomeAssistantUpdate, run_id)
run.state, run.message, run.finished_at = state, message, datetime.now(timezone.utc)
# The page should show the new versions now, not at the next poll.
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if connection is not None:
connection.last_polled_at = None
try:
async with session.begin_nested():
await update_entry(session, run)
except Exception:
logger.exception("Home Assistant update %s: its timeline entry could not be written", run_id)
async def advance(app, transport=None, connect=None) -> None:
"""Move the update under way one step on, if there is one."""
async with app.db_sessionmaker() as session:
run = await under_way(session)
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
if run is None:
return
if connection is None:
await _finish(app, run.id, "failed", "The Home Assistant connection was removed.")
return
try:
if run.state == "backing_up":
await _advance_backup(app, run, connection, transport)
elif run.state == "installing":
await _advance_install(app, run, connection, transport)
else:
await _advance_reboot(app, run, connection, transport, connect)
except client.Refused as exc:
await _mark_refused(app, connection.id, str(exc))
await _finish(app, run.id, "failed", str(exc))
except client.Rejected as exc:
await _finish(app, run.id, "failed", f"Home Assistant wouldn't install it: {exc}")
except client.Unreachable as exc:
# Expected while Home Assistant restarts: the next tick asks again.
logger.info("Home Assistant update %s: %s", run.id, exc)
await _past_deadline(app, run)
def _elapsed(since: datetime) -> float:
return (datetime.now(timezone.utc) - since).total_seconds()
async def _past_deadline(app, run) -> bool:
if run.state == "backing_up" and _elapsed(run.started_at) > BACKUP_DEADLINE_S:
await _finish(app, run.id, "failed",
f"No new backup within {BACKUP_DEADLINE_S // 60} minutes. Nothing was installed.")
return True
if run.state == "installing":
limit = INSTALL_DEADLINE_S.get(run.kind, INSTALL_DEADLINE_S["os"])
if _elapsed(run.install_sent_at or run.backup_at or run.started_at) > limit:
await _finish(app, run.id, "failed",
f"Still not on {run.to_version} after {limit // 60} minutes. Look in Home Assistant.")
return True
if run.state == "rebooting" and _elapsed(run.reboot_sent_at or run.install_sent_at or run.started_at) > REBOOT_DEADLINE_S:
await _finish(app, run.id, "failed", f"{run.to_version} was installed, but isn't running "
f"{REBOOT_DEADLINE_S // 60} minutes after the reboot. Look in Home Assistant.")
return True
return False
async def _advance_backup(app, run, connection, transport) -> None:
token = await client.access_token(connection, transport)
url, verify = connection.url, connection.verify_tls
backup = await client.read_state(url, LAST_BACKUP, token, verify, transport)
value = (backup or {}).get("state")
if is_newer(value, run.backup_before):
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantUpdate, run.id)
row.state, row.backup_at = "installing", parse_moment(value)
return
event = await client.read_state(url, BACKUP_EVENT, token, verify, transport)
if event and event.get("state") != run.backup_event_before \
and (event.get("attributes") or {}).get("event_type") == "failed":
reason = (event.get("attributes") or {}).get("failed_reason")
await _finish(app, run.id, "failed",
f"The backup failed{f' ({reason})' if reason else ''}. Nothing was installed.")
return
await _past_deadline(app, run)
async def _mark_sent(app, run_id: str, at: datetime | None) -> None:
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantUpdate, run_id)
row.install_sent_at = at
async def _advance_install(app, run, connection, transport) -> None:
token = await client.access_token(connection, transport)
url, verify = connection.url, connection.verify_tls
if run.install_sent_at is None:
# Marked first: Steward stopping here leaves an install that runs out its
# deadline, where asking twice would fail one that is going fine.
await _mark_sent(app, run.id, datetime.now(timezone.utc))
try:
await client.call_action(url, "update", "install", {"entity_id": run.entity_id}, token, verify, transport)
except client.Unreachable:
await _mark_sent(app, run.id, None)
raise
return
current = await client.read_state(url, run.entity_id, token, verify, transport)
attrs = (current or {}).get("attributes") or {}
installed = attrs.get("installed_version")
if installed and (installed == run.to_version
or (current.get("state") == "off" and installed != run.from_version)):
if run.reboot_after:
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantUpdate, run.id)
row.state = "rebooting"
return
await _finish(app, run.id, "done", None)
return
await _past_deadline(app, run)
async def _mark_reboot_sent(app, run_id: str, at: datetime | None) -> None:
async with app.db_sessionmaker() as session:
async with session.begin():
row = await session.get(HomeAssistantUpdate, run_id)
row.reboot_sent_at = at
async def _advance_reboot(app, run, connection, transport, connect) -> None:
"""Reboot once, and wait for the new version to be what's running. The version
is judged against the old one, so a machine that fell back to it fails at the
deadline, as does one that never comes back."""
token = await client.access_token(connection, transport)
url, verify = connection.url, connection.verify_tls
if run.reboot_sent_at is None:
await _mark_reboot_sent(app, run.id, datetime.now(timezone.utc))
try:
await client.call_action(url, "hassio", "host_reboot", {}, token, verify, transport)
except client.Unreachable:
await _mark_reboot_sent(app, run.id, None)
await _past_deadline(app, run)
return
except client.Rejected as exc:
await _finish(app, run.id, "failed", f"{run.to_version} was installed, but Home Assistant "
f"wouldn't reboot: {exc} It needs a reboot to take effect.")
return
found = await client.read_os(url, token, verify, connect)
if found and found.get("version") not in (None, run.from_version) and not found.get("version_pending"):
await _finish(app, run.id, "done", None)
return
await _past_deadline(app, run)
+4
View File
@@ -29,6 +29,10 @@ def setup(app: "Quart") -> None:
from steward.core.alerts import names_behind from steward.core.alerts import names_behind
from .routes import name_hottest_sensor from .routes import name_hottest_sensor
names_behind("host_agent", "temp_c_max", name_hottest_sensor) names_behind("host_agent", "temp_c_max", name_hottest_sensor)
# get_host over MCP marks the sensors the host's temperature leaves out.
from steward.mcp.tools.hosts import judge_sensors
from .routes import split_temps
judge_sensors(lambda host, temps: {k: why for k, (_c, why) in split_temps(host, temps)[1].items()})
# A reboot the agent announced is silence with a reason, for a while (#5756). # A reboot the agent announced is silence with a reason, for a while (#5756).
from datetime import datetime, timezone from datetime import datetime, timezone
from steward.core.alerts import agents_away from steward.core.alerts import agents_away
+5 -3
View File
@@ -22,7 +22,7 @@ from collections import deque
from concurrent.futures import ThreadPoolExecutor, wait as wait_futures from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
from datetime import datetime, timezone from datetime import datetime, timezone
AGENT_VERSION = "1.26.0" AGENT_VERSION = "1.27.0"
# Default path to the local Docker Engine socket. Overridable via the # Default path to the local Docker Engine socket. Overridable via the
# `docker_socket` config key; collection is silently skipped if it's absent or # `docker_socket` config key; collection is silently skipped if it's absent or
@@ -480,7 +480,9 @@ def nfs_delays(cur: dict, prev: dict, dt: float) -> list:
def collect_temps() -> list: def collect_temps() -> list:
"""Return [{label, celsius}] from /sys/class/hwmon (best-effort, may be empty).""" """Return [{label, celsius, chip}] from /sys/class/hwmon (best-effort, may
be empty). `chip` is the hwmon driver's name (coretemp, nvme, nct6798…),
which says what a bare label like "Sensor 2" is measuring."""
out: list = [] out: list = []
try: try:
chips = os.listdir(HWMON_DIR) chips = os.listdir(HWMON_DIR)
@@ -511,7 +513,7 @@ def collect_temps() -> list:
except OSError: except OSError:
pass pass
name = label or (f"{chip}_{idx}" if chip else idx) name = label or (f"{chip}_{idx}" if chip else idx)
out.append({"label": name, "celsius": round(milli / 1000.0, 1)}) out.append({"label": name, "celsius": round(milli / 1000.0, 1), "chip": chip})
return out return out
+3 -61
View File
@@ -21,11 +21,12 @@ the host it runs on, picks up where it was.
from __future__ import annotations from __future__ import annotations
import json import json
from datetime import datetime, time, timedelta, timezone from datetime import datetime, timedelta
from steward.core.weekly import WEEKDAYS, describe, fired_slot, is_due, last_slot, next_slot, parse_at # noqa: F401
from . import maintenance, rebalance, timeline from . import maintenance, rebalance, timeline
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
# How long a host may take to be back in service after its job finished. # How long a host may take to be back in service after its job finished.
SETTLE_LIMIT = timedelta(minutes=30) SETTLE_LIMIT = timedelta(minutes=30)
# An agent heard from this recently is posting. # An agent heard from this recently is posting.
@@ -33,58 +34,6 @@ POSTING_WITHIN = timedelta(minutes=3)
RUN_STATES = ("running", "done", "failed", "stopped") RUN_STATES = ("running", "done", "failed", "stopped")
def _zone(name: str):
from steward.core.formatting import zone
return zone(name)
def parse_at(at_time: str) -> time:
h, m = (int(x) for x in at_time.split(":", 1))
if not (0 <= h < 24 and 0 <= m < 60):
raise ValueError(at_time)
return time(h, m)
def last_slot(plan, now: datetime) -> datetime:
"""The most recent scheduled time at or before `now`, in UTC."""
tz = _zone(plan.time_zone)
local = now.astimezone(tz)
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
if slot > local:
slot -= timedelta(days=7)
return slot.astimezone(timezone.utc)
def fired_slot(plan) -> datetime | None:
"""The last slot that fired, or None while none has. A new plan's
last_slot_at holds the moment it was made, so its first slot is the
next one rather than one already past; that moment is no slot (#5863)."""
at = plan.last_slot_at
if at is None:
return None
return at if last_slot(plan, at) == at else None
def next_slot(plan, now: datetime) -> datetime:
return last_slot(plan, now) + timedelta(days=7)
def is_due(plan, now: datetime) -> datetime | None:
"""The slot to fire now, or None. A slot fires once (last_slot_at), and
only while its window is still open: a Steward that was down over the slot
starts late, not a day late."""
if not plan.enabled:
return None
slot = last_slot(plan, now)
since = plan.last_slot_at or plan.created_at
if since is not None and slot <= since:
return None
if now - slot >= timedelta(minutes=plan.window_minutes):
return None
return slot
def back_in_service(*, last_seen_at, node, services, job_finished_at, now: datetime, def back_in_service(*, last_seen_at, node, services, job_finished_at, now: datetime,
expect_active: bool = True) -> str | None: expect_active: bool = True) -> str | None:
"""None when the host is back in service, else why not yet. `node` is the """None when the host is back in service, else why not yet. `node` is the
@@ -134,13 +83,6 @@ def next_move(run, plan, job, back: str | None, now: datetime) -> tuple[str, str
return "settling", back return "settling", back
def describe(plan) -> str:
""""Sundays at 03:00 (Europe/London), up to 4 h" for the page."""
hours = plan.window_minutes / 60
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
# ── applied to the database: the minute tick, Run now, the page ────────────── # ── applied to the database: the minute tick, Run now, the page ──────────────
async def group_hosts(session, group_id: str) -> list: async def group_hosts(session, group_id: str) -> list:
+7 -5
View File
@@ -101,10 +101,10 @@ def valid(req) -> bool:
def run(cmd: list, deadline: int = COMMAND_DEADLINE_SECONDS, env: dict | None = None, def run(cmd: list, deadline: int = COMMAND_DEADLINE_SECONDS, env: dict | None = None,
ok_codes: tuple = (0,), keep: int = OUTPUT_KEPT_CHARS): ok_codes: tuple = (0,), keep: int | None = OUTPUT_KEPT_CHARS):
"""(ok, output tail) of a command, bounded by `deadline`. `ok_codes` are """(ok, output tail) of a command, bounded by `deadline`. `ok_codes` are
the exit codes that mean it worked (zypper says more than 0); `keep` is the exit codes that mean it worked (zypper says more than 0); `keep` is
how much of the output's end is kept.""" how much of the output's end is kept, None for all of it."""
try: try:
done = subprocess.run(cmd, capture_output=True, text=True, timeout=deadline, done = subprocess.run(cmd, capture_output=True, text=True, timeout=deadline,
env=dict(os.environ, LANG="C", LC_ALL="C", **(env or {}))) env=dict(os.environ, LANG="C", LC_ALL="C", **(env or {})))
@@ -113,7 +113,7 @@ def run(cmd: list, deadline: int = COMMAND_DEADLINE_SECONDS, env: dict | None =
except OSError as exc: except OSError as exc:
return False, str(exc) return False, str(exc)
out = (done.stdout or "") + (done.stderr or "") out = (done.stdout or "") + (done.stderr or "")
return done.returncode in ok_codes, out[-keep:] return done.returncode in ok_codes, out if keep is None else out[-keep:]
REBOOT_MARKER = "/var/run/reboot-required" # Debian and Ubuntu REBOOT_MARKER = "/var/run/reboot-required" # Debian and Ubuntu
@@ -226,9 +226,11 @@ def do_update() -> dict:
"(apt, dnf, yum, zypper, pacman) on this host"} "(apt, dnf, yum, zypper, pacman) on this host"}
out = "" out = ""
for label, cmd, deadline, env, ok_codes in update_steps(manager): for label, cmd, deadline, env, ok_codes in update_steps(manager):
ok, out = run(cmd, deadline=deadline, env=env, ok_codes=ok_codes) # The whole output: every manager prints its count before the long
# log of what it then does, so a tail loses it on a large update.
ok, out = run(cmd, deadline=deadline, env=env, ok_codes=ok_codes, keep=None)
if not ok: if not ok:
return {"ok": False, "detail": f"{label} failed: " + out} return {"ok": False, "detail": f"{label} failed: " + out[-OUTPUT_KEPT_CHARS:]}
changed = upgraded_count(out) changed = upgraded_count(out)
if manager == "apt": if manager == "apt":
run(["apt-get", "-y", "autoremove"], env={"DEBIAN_FRONTEND": "noninteractive"}) run(["apt-get", "-y", "autoremove"], env={"DEBIAN_FRONTEND": "noninteractive"})
+19 -7
View File
@@ -124,13 +124,15 @@ def is_thermometer(host_name: str, label: str, celsius: float) -> bool:
def hottest_sensor(host_name: str, temps: dict[str, float]) -> list[str]: def hottest_sensor(host_name: str, temps: dict[str, float]) -> list[str]:
"""Which sensor sets a host's temperature, as an alert on it says: """Which sensor sets a host's temperature, as an alert on it says:
["Package id 0 at 86°C"], or [] with no thermometer among `temps`.""" ["CPU · Package id 0 at 86°C"], or [] with no thermometer among `temps`.
The chip leads the label when the host's agent has said which it is."""
from steward.core.formatting import metric_value from steward.core.formatting import metric_value
real = {label: c for label, c in temps.items() if is_thermometer(host_name, label, c)} real = {label: c for label, c in temps.items() if is_thermometer(host_name, label, c)}
if not real: if not real:
return [] return []
from .sensors import sensor_name
label = max(real, key=real.get) label = max(real, key=real.get)
return [f"{label} at {metric_value('temp_c', real[label])}"] return [f"{sensor_name(host_name, label)} at {metric_value('temp_c', real[label])}"]
async def name_hottest_sensor(session, resource: str) -> list[str]: async def name_hottest_sensor(session, resource: str) -> list[str]:
@@ -374,6 +376,8 @@ def _expand_sample_to_metrics(
# counts only readings that can be real (#5759), from sensors that behave # counts only readings that can be real (#5759), from sensors that behave
# like thermometers (#5783). # like thermometers (#5783).
max_temp: float | None = None max_temp: float | None = None
from .sensors import note_chips
note_chips(host_name, sample.get("temps"))
for t in sample.get("temps") or []: for t in sample.get("temps") or []:
c = t.get("celsius") c = t.get("celsius")
if c is None: if c is None:
@@ -1003,7 +1007,7 @@ def _new_token_pair() -> tuple[str, str]:
@host_agent_bp.get("/vitals/<host_id>") @host_agent_bp.get("/vitals/<host_id>")
@require_role(UserRole.viewer) @require_role(UserRole.viewer)
async def host_vitals(host_id: str): async def host_vitals(host_id: str):
"""Compact live vitals strip (CPU/MEM/DISK/LOAD + pressure) for the Hosts hub. """Compact live vitals strip (CPU/MEM/DISK/LOAD/temperature + pressure) for the Hosts hub.
Self-contained fragment, polled by hosts/detail.html. Renders nothing until Self-contained fragment, polled by hosts/detail.html. Renders nothing until
the agent reports — the Agent panel below then carries provisioning. the agent reports — the Agent panel below then carries provisioning.
@@ -1018,15 +1022,16 @@ async def host_vitals(host_id: str):
latest = await _latest_metrics_for_host(db, host.name) if reg else {} latest = await _latest_metrics_for_host(db, host.name) if reg else {}
norms = await host_svc.host_norms(db, host.name) if reg else {} norms = await host_svc.host_norms(db, host.name) if reg else {}
# The last 6 h for the at-a-glance sparklines (cpu/mem/load host-level, # The last 6 h for the at-a-glance sparklines (cpu/mem/load/temp host-level,
# disk from the root mount), averaged into ~120 points in SQL, so the # disk from the root mount), averaged into ~120 points in SQL, so the
# line spans the whole window and only what is drawn is fetched. # line spans the whole window and only what is drawn is fetched.
series_raw: dict[str, list[float]] = {"cpu": [], "mem": [], "disk": [], "load": []} series_raw: dict[str, list[float]] = {"cpu": [], "mem": [], "disk": [], "load": [], "temp": []}
if reg: if reg:
since = datetime.now(timezone.utc) - timedelta(hours=6) since = datetime.now(timezone.utc) - timedelta(hours=6)
for key, resource, metric in (("cpu", host.name, "cpu_pct"), for key, resource, metric in (("cpu", host.name, "cpu_pct"),
("mem", host.name, "mem_used_pct"), ("mem", host.name, "mem_used_pct"),
("load", host.name, "load_1m"), ("load", host.name, "load_1m"),
("temp", host.name, "temp_c_max"),
("disk", host.name + ":/", "disk_used_pct")): ("disk", host.name + ":/", "disk_used_pct")):
series_raw[key] = [v for _t, v in await metric_points( series_raw[key] = [v for _t, v in await metric_points(
db, SOURCE_MODULE, resource, metric, since, target=120)] db, SOURCE_MODULE, resource, metric, since, target=120)]
@@ -1053,9 +1058,16 @@ async def host_vitals(host_id: str):
reporting = bool(reg) and ls is not None reporting = bool(reg) and ls is not None
stale = reporting and (datetime.now(timezone.utc) - ls).total_seconds() > _stale_after_seconds() stale = reporting and (datetime.now(timezone.utc) - ls).total_seconds() > _stale_after_seconds()
cpu, mem = hostlvl.get("cpu_pct"), hostlvl.get("mem_used_pct") cpu, mem = hostlvl.get("cpu_pct"), hostlvl.get("mem_used_pct")
# The hottest sensor that counts (temp_c_max leaves out unwired inputs),
# named so the tile says what it is reading.
temp = hostlvl.get("temp_c_max")
prefix = host.name + ":temp:"
hottest = hottest_sensor(host.name, {r[len(prefix):]: m["temp_c"] for r, m in latest.items()
if r.startswith(prefix) and m.get("temp_c") is not None})
thresholds = current_app.config.get("THRESHOLDS", {}) thresholds = current_app.config.get("THRESHOLDS", {})
worst = worst_reading((("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", disk_root, "disk"), worst = worst_reading((("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", disk_root, "disk"),
("Inodes on /", inodes_root, "disk"), ("Load per core", load_per_core, "load")), ("Inodes on /", inodes_root, "disk"), ("Load per core", load_per_core, "load"),
("Temperature", temp, "temp")),
thresholds, host_svc.host_normals(norms, thresholds)) thresholds, host_svc.host_normals(norms, thresholds))
# Each tile's grade; CPU, memory and load within the host's own normal # Each tile's grade; CPU, memory and load within the host's own normal
# for the hour read healthy, and say what is usual (#5742). # for the hour read healthy, and say what is usual (#5742).
@@ -1068,7 +1080,7 @@ async def host_vitals(host_id: str):
return await render_template( return await render_template(
"_host_vitals.html", "_host_vitals.html",
reg=reg, cpu=cpu, mem=mem, disk_root=disk_root, load_per_core=load_per_core, reg=reg, cpu=cpu, mem=mem, disk_root=disk_root, load_per_core=load_per_core,
inodes_root=inodes_root, inodes_root=inodes_root, temp=temp, hottest=hottest[0].rsplit(" at ", 1)[0] if hottest else None,
sparks=sparks, psi=psi, reporting=reporting, stale=stale, worst=worst, sparks=sparks, psi=psi, reporting=reporting, stale=stale, worst=worst,
grades=grades, usual=usual, grades=grades, usual=usual,
clock=clock_words(reg.clock_offset_s if reg else None, current_app.config.get("THRESHOLDS", {})), clock=clock_words(reg.clock_offset_s if reg else None, current_app.config.get("THRESHOLDS", {})),
+1 -1
View File
@@ -2,7 +2,7 @@
ranked against the host's own total over a window. ranked against the host's own total over a window.
A host's net_tx_bps (net_rx_bps) is what its physical interfaces carried; each A host's net_tx_bps (net_rx_bps) is what its physical interfaces carried; each
container's comes from Docker ingest (resource "<host>/<container>") and each container's comes from Docker ingest (resource "<host>/<service or container>") and each
NFS mount's from the agent (resource "<host>:<mount>"). All three are read in NFS mount's from the agent (resource "<host>:<mount>"). All three are read in
the same time buckets, so the busiest bucket of the host's total can be put the same time buckets, so the busiest bucket of the host's total can be put
down to who was sending in it. What no container or mount accounts for is the down to who was sending in it. What no container or mount accounts for is the
+45
View File
@@ -53,6 +53,51 @@ def unwired(host_name: str) -> dict[str, str]:
return _unwired.get(host_name, {}) return _unwired.get(host_name, {})
# {host name: {sensor label: hwmon chip}}, as the host's latest sample said.
# A sensor's metric is keyed by its label alone; the chip only names it.
_chips: dict[str, dict[str, str]] = {}
# What each hwmon driver measures, in the words a page uses. A driver not
# listed is shown by its own name.
_CHIP_WORDS = (
(re.compile(r"^(coretemp|k10temp|zenpower|cpu_thermal)$"), "CPU"),
(re.compile(r"^nvme$"), "NVMe"),
(re.compile(r"^drivetemp$"), "Drive"),
(re.compile(r"^(nct\d+|it\d+|w83\w+|f71\w+)$"), "Board"),
(re.compile(r"^pch_\w+$"), "Chipset"),
(re.compile(r"^(amdgpu|radeon|nouveau|i915|xe)$"), "GPU"),
(re.compile(r"^(spd5118|jc42)$"), "Memory"),
(re.compile(r"^iwlwifi"), "Wi-Fi"),
(re.compile(r"^acpitz$"), "ACPI"),
)
def note_chips(host_name: str, temps) -> None:
"""Remember which chip each of a host's sensors is on, from a sample's
temps. An agent that doesn't say leaves what is known as it is."""
seen = {str(t["label"]): str(t["chip"]) for t in temps or [] if t.get("chip")}
if seen:
_chips[host_name] = seen
def chip_word(chip: str) -> str:
"""What a hwmon chip measures ("nvme" → "NVMe"), or its own name."""
for pattern, word in _CHIP_WORDS:
if pattern.match(chip):
return word
return chip
def sensor_name(host_name: str, label: str) -> str:
"""A sensor as a page names it: its chip, then its label, as in
"NVMe · Sensor 2". The label alone when the chip isn't known or the
label already carries it ("nvme_temp1")."""
chip = _chips.get(host_name, {}).get(label)
if not chip or label.lower().startswith(chip.lower()):
return label
return f"{chip_word(chip)} · {label}"
def reference_label(labels) -> str | None: def reference_label(labels) -> str | None:
"""The sensor that reads the host's CPU, or None when it has none.""" """The sensor that reads the host's CPU, or None when it has none."""
labels = sorted(labels) labels = sorted(labels)
+10 -3
View File
@@ -1,10 +1,11 @@
{# Host health header — live polled into the host hub: live or stale, the worst {# Host health header — live polled into the host hub: live or stale, the worst
reading against Settings → Thresholds, last seen, then CPU / memory / disk / reading against Settings → Thresholds, last seen, then CPU / memory / disk /
load tiles graded by the same cutoffs (`grades`; CPU, memory and load within load tiles, and the hottest sensor that counts with its name, graded by the same cutoffs (`grades`; CPU, memory and load within
the host's own normal for the hour read healthy, and `usual` says so). Renders nothing until the agent the host's own normal for the hour read healthy, and `usual` says so). Renders nothing until the agent
reports (the Agent panel below then shows provisioning). #} reports (the Agent panel below then shows provisioning). #}
{% from "_macros.html" import status_badge, stat_tile %} {% from "_macros.html" import status_badge, stat_tile %}
{% macro pct(v) %}{{ '%.0f%%'|format(v) if v is not none else '—' }}{% endmacro %} {% macro pct(v) %}{{ '%.0f%%'|format(v) if v is not none else '—' }}{% endmacro %}
{% macro deg(v) %}{{ '%.0f°C'|format(v) if v is not none else '—' }}{% endmacro %}
{% set grades = grades or {} %} {% set grades = grades or {} %}
{% if reporting %} {% if reporting %}
<section class="card host-health" aria-label="Host health"> <section class="card host-health" aria-label="Host health">
@@ -12,7 +13,7 @@
{{ status_badge('stale' if stale else 'up', 'Stale' if stale else 'Live') }} {{ status_badge('stale' if stale else 'up', 'Stale' if stale else 'Live') }}
<span class="host-health-summary"> <span class="host-health-summary">
{%- if stale %}No report since {{ reg.last_seen_at|ago }}; the readings below are from then. {%- if stale %}No report since {{ reg.last_seen_at|ago }}; the readings below are from then.
{%- elif worst %}<span class="tone-{{ worst[2] }}">{{ worst[0] }} {{ pct(worst[1]) }}</span> is past its {{ "critical" if worst[2] == "crit" else "warning" }} threshold. {%- elif worst %}<span class="tone-{{ worst[2] }}">{{ worst[0] }} {{ deg(worst[1]) if worst[0] == "Temperature" else pct(worst[1]) }}</span> is past its {{ "critical" if worst[2] == "crit" else "warning" }} threshold.
{%- elif usual %}Every reading is within its thresholds or its usual level for this hour. {%- elif usual %}Every reading is within its thresholds or its usual level for this hour.
{%- else %}Every reading is within its thresholds.{% endif -%} {%- else %}Every reading is within its thresholds.{% endif -%}
</span> </span>
@@ -21,14 +22,20 @@
{%- if reg.agent_version %}{{ " · " if not stale }}agent v{{ reg.agent_version }}{% endif -%} {%- if reg.agent_version %}{{ " · " if not stale }}agent v{{ reg.agent_version }}{% endif -%}
{%- if clock and clock[0] %} · <span{% if clock[1] in ('warn', 'crit') %} class="tone-{{ clock[1] }}" title="Further from Steward's clock than Settings → Thresholds allows: check what keeps this host's time in sync"{% endif %}>{{ clock[0] }}</span>{% endif -%} {%- if clock and clock[0] %} · <span{% if clock[1] in ('warn', 'crit') %} class="tone-{{ clock[1] }}" title="Further from Steward's clock than Settings → Thresholds allows: check what keeps this host's time in sync"{% endif %}>{{ clock[0] }}</span>{% endif -%}
</span> </span>
{# Charts, every mount, interface, disk and temperature sensor. #}
<a href="/plugins/host_agent/{{ reg.host_id }}/" class="btn btn-sm btn-ghost" style="margin-left:auto;">Full metrics →</a>
</div> </div>
<div class="stat-strip host-health-tiles" style="margin-bottom:0;"> <div class="stat-strip host-health-tiles" style="margin-bottom:0;--tiles:{{ 5 if temp is not none else 4 }};">
{{ stat_tile("CPU", pct(cpu), state=(grades.cpu or [status_class(cpu, 'cpu')])[0], hint=sparks.cpu|safe) }} {{ stat_tile("CPU", pct(cpu), state=(grades.cpu or [status_class(cpu, 'cpu')])[0], hint=sparks.cpu|safe) }}
{{ stat_tile("Memory", pct(mem), state=(grades.mem or [status_class(mem, 'mem')])[0], hint=sparks.mem|safe) }} {{ stat_tile("Memory", pct(mem), state=(grades.mem or [status_class(mem, 'mem')])[0], hint=sparks.mem|safe) }}
{%- set inodes_worse = inodes_root is defined and inodes_root is not none and (disk_root is none or inodes_root > disk_root) %} {%- set inodes_worse = inodes_root is defined and inodes_root is not none and (disk_root is none or inodes_root > disk_root) %}
{{ stat_tile("Disk /", pct(disk_root) ~ (" · inodes " ~ pct(inodes_root) if inodes_worse else ""), {{ stat_tile("Disk /", pct(disk_root) ~ (" · inodes " ~ pct(inodes_root) if inodes_worse else ""),
state=(grades.disk or [status_class(disk_root, 'disk')])[0], hint=sparks.disk|safe) }} state=(grades.disk or [status_class(disk_root, 'disk')])[0], hint=sparks.disk|safe) }}
{{ stat_tile("Load per core", pct(load_per_core), state=(grades.load or [status_class(load_per_core, 'load')])[0], hint=sparks.load|safe) }} {{ stat_tile("Load per core", pct(load_per_core), state=(grades.load or [status_class(load_per_core, 'load')])[0], hint=sparks.load|safe) }}
{% if temp is not none %}
{%- set temp_hint %}{{ sparks.temp|safe }}{% if hottest %}<span class="stat-tile-sensor" title="{{ hottest }}">{{ hottest }}</span>{% endif %}{% endset %}
{{ stat_tile("Temperature", deg(temp), state=status_class(temp, 'temp'), hint=temp_hint) }}
{% endif %}
</div> </div>
{% if usual and not stale %} {% if usual and not stale %}
<p class="field-hint" style="margin-top:0.6rem;">{{ usual | join("; ") | capitalize }}.</p> <p class="field-hint" style="margin-top:0.6rem;">{{ usual | join("; ") | capitalize }}.</p>
+3 -4
View File
@@ -14,10 +14,9 @@
</div> </div>
{% if reporting %} {% if reporting %}
{# Reporting: live vitals are shown by the vitals strip above; this panel is {# Reporting: live vitals, and the way to the full metrics, are in the
the way to the full metrics, with the lifecycle actions folded under vitals strip at the top of the page; this panel holds what the host has
"Manage agent" (#5278). #} waiting, with the lifecycle actions folded under "Manage agent" (#5278). #}
<a href="/plugins/host_agent/{{ host.id }}/" class="btn btn-sm btn-ghost">Full metrics →</a>
{# What the host has waiting, from the agent's hourly check (#5765): a {# What the host has waiting, from the agent's hourly check (#5765): a
needed reboot first, then the updates. #} needed reboot first, then the updates. #}
@@ -0,0 +1,23 @@
"""A target whose token was refused stops being scraped until it is saved
Revision ID: prometheus_003_refused
Revises: prometheus_002_templates
Create Date: 2026-11-01
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "prometheus_003_refused"
down_revision: Union[str, None] = "prometheus_002_templates"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("prometheus_targets", sa.Column("refused", sa.Boolean, nullable=False, server_default=sa.false()))
def downgrade() -> None:
op.drop_column("prometheus_targets", "refused")
@@ -0,0 +1,23 @@
"""A target can take its token from another plugin's sign-in
Revision ID: prometheus_004_token_from
Revises: prometheus_003_refused
Create Date: 2026-11-01
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "prometheus_004_token_from"
down_revision: Union[str, None] = "prometheus_003_refused"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True))
def downgrade() -> None:
op.drop_column("prometheus_targets", "token_from")
@@ -0,0 +1,32 @@
"""Every target holds its own token: one that took it from a sign-in goes
Revision ID: prometheus_005_own_tokens
Revises: prometheus_004_token_from
Create Date: 2026-11-01
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "prometheus_005_own_tokens"
down_revision: Union[str, None] = "prometheus_004_token_from"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Left in place, such a target would ask with no token, and some services ban
# an address for that. Its newest values go with it; history ages out. On a new
# database the metric tables may not exist yet, and there is nothing to remove.
if sa.inspect(op.get_bind()).has_table("plugin_metric_latest"):
op.execute("""
DELETE FROM plugin_metric_latest l USING prometheus_targets t
WHERE t.token_from IS NOT NULL AND l.source_module = 'prometheus'
AND (l.resource_name = t.name OR left(l.resource_name, length(t.name) + 1) = t.name || ':')""")
op.execute("DELETE FROM prometheus_targets WHERE token_from IS NOT NULL")
op.drop_column("prometheus_targets", "token_from")
def downgrade() -> None:
op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True))
+3 -1
View File
@@ -17,7 +17,8 @@ class PrometheusTarget(Base):
patterns ("fabledarchives_*") an admin chooses: Steward does the patterns ("fabledarchives_*") an admin chooses: Steward does the
filtering, the endpoint serves everything. The last_* columns are the filtering, the endpoint serves everything. The last_* columns are the
newest scrape's outcome, written by the scraper, so a page can say why a newest scrape's outcome, written by the scraper, so a page can say why a
target isn't answering without asking it. target isn't answering without asking it. `refused` says its token was
refused, which stops the scraping until the target is saved or resumed.
""" """
__tablename__ = "prometheus_targets" __tablename__ = "prometheus_targets"
@@ -31,6 +32,7 @@ class PrometheusTarget(Base):
drop: Mapped[list] = mapped_column(JSON, nullable=False, default=list) drop: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
series_cap: Mapped[int] = mapped_column(Integer, nullable=False, default=1000) series_cap: Mapped[int] = mapped_column(Integer, nullable=False, default=1000)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
refused: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc)) DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
last_scraped_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) last_scraped_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+2
View File
@@ -153,6 +153,7 @@ async def save_target(target_id: str):
# Where it is or what is kept may have changed: the last outcome no # Where it is or what is kept may have changed: the last outcome no
# longer describes it, so the page says "not scraped yet" until the next scrape. # longer describes it, so the page says "not scraped yet" until the next scrape.
target.last_scraped_at = target.last_ok_at = target.last_error = None target.last_scraped_at = target.last_ok_at = target.last_error = None
target.refused = False
target.last_kept = target.last_dropped = target.last_capped = 0 target.last_kept = target.last_dropped = target.last_capped = 0
if old_name != target.name: if old_name != target.name:
# Its history is stored under its name, and its labelled series # Its history is stored under its name, and its labelled series
@@ -188,6 +189,7 @@ async def toggle_target(target_id: str):
# Resumed: it reads "not scraped yet" until the next scrape, # Resumed: it reads "not scraped yet" until the next scrape,
# rather than stale from before the pause. # rather than stale from before the pause.
target.last_scraped_at = target.last_error = None target.last_scraped_at = target.last_error = None
target.refused = False
enabled, name = target.enabled, target.name enabled, name = target.enabled, target.name
# A rate needs two readings in a row; one from before the pause isn't one. # A rate needs two readings in a row; one from before the pause isn't one.
from . import scheduler from . import scheduler
+3 -2
View File
@@ -40,7 +40,7 @@ def make_scrape_task(app: "Quart") -> ScheduledTask:
def is_due(target, now: datetime) -> bool: def is_due(target, now: datetime) -> bool:
"""Whether a target's interval has elapsed since its last scrape, with """Whether a target's interval has elapsed since its last scrape, with
half a tick of slack so tick jitter doesn't push it back a whole tick.""" half a tick of slack so tick jitter doesn't push it back a whole tick."""
if not target.enabled: if not target.enabled or target.refused:
return False return False
if target.last_scraped_at is None: if target.last_scraped_at is None:
return True return True
@@ -115,7 +115,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None:
from steward.core.alerts import MetricBatch from steward.core.alerts import MetricBatch
from .models import PrometheusTarget from .models import PrometheusTarget
from .scraper import ScrapeError from .scraper import ScrapeError, TokenRefused
from .service import (IS_UP, OWN_METRICS, SCRAPE_MS, SERIES_CAPPED, SERIES_KEPT, SOURCE, from .service import (IS_UP, OWN_METRICS, SCRAPE_MS, SERIES_CAPPED, SERIES_KEPT, SOURCE,
map_families) map_families)
@@ -132,6 +132,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None:
if not isinstance(result, ScrapeError): if not isinstance(result, ScrapeError):
logger.warning("Prometheus scrape of %s failed: %r", target.name, result) logger.warning("Prometheus scrape of %s failed: %r", target.name, result)
row.last_error = str(result) or "The scrape failed." row.last_error = str(result) or "The scrape failed."
row.refused = isinstance(result, TokenRefused)
batch.add(source_module=SOURCE, resource_name=row.name, metric_name=IS_UP, batch.add(source_module=SOURCE, resource_name=row.name, metric_name=IS_UP,
value=0.0, recorded_at=now) value=0.0, recorded_at=now)
continue continue
+7 -2
View File
@@ -18,6 +18,11 @@ class ScrapeError(Exception):
"""Why a scrape got nothing, said to the person.""" """Why a scrape got nothing, said to the person."""
class TokenRefused(ScrapeError):
"""The endpoint refused the token. Asking again won't change that, and
Home Assistant, for one, bans an address after a few refusals."""
async def scrape(url: str, token: str = "", verify_tls: bool = True, async def scrape(url: str, token: str = "", verify_tls: bool = True,
transport: httpx.AsyncBaseTransport | None = None) -> dict[str, prometheus.Family]: transport: httpx.AsyncBaseTransport | None = None) -> dict[str, prometheus.Family]:
"""The endpoint's families. `transport` stands in for the network in tests.""" """The endpoint's families. `transport` stands in for the network in tests."""
@@ -29,8 +34,8 @@ async def scrape(url: str, token: str = "", verify_tls: bool = True,
transport=transport, follow_redirects=False) as client: transport=transport, follow_redirects=False) as client:
async with client.stream("GET", url, headers=headers) as response: async with client.stream("GET", url, headers=headers) as response:
if response.status_code in (401, 403): if response.status_code in (401, 403):
raise ScrapeError(f"Answered {response.status_code}: the token was refused. " raise TokenRefused(f"Answered {response.status_code}: the token was refused. "
"Check the token, or make a new one on the service.") "Check the token, or make a new one on the service.")
if response.status_code != 200: if response.status_code != 200:
raise ScrapeError(f"Answered {response.status_code} {response.reason_phrase}, not metrics.") raise ScrapeError(f"Answered {response.status_code} {response.reason_phrase}, not metrics.")
body = bytearray() body = bytearray()
+3
View File
@@ -271,6 +271,9 @@ class TargetState:
def target_state(target: PrometheusTarget, now: datetime) -> TargetState: def target_state(target: PrometheusTarget, now: datetime) -> TargetState:
if not target.enabled: if not target.enabled:
return TargetState("paused", "Paused: not scraped until it's resumed.") return TargetState("paused", "Paused: not scraped until it's resumed.")
if target.refused:
return TargetState("down", f"{target.last_error} Steward stopped asking, as some services ban an "
"address after a few refusals. Save a new token to start again.")
if target.last_scraped_at is None: if target.last_scraped_at is None:
return TargetState("pending", "Not scraped yet.") return TargetState("pending", "Not scraped yet.")
if (now - target.last_scraped_at).total_seconds() > STALE_SCRAPES * max(target.interval_seconds, 1): if (now - target.last_scraped_at).total_seconds() > STALE_SCRAPES * max(target.interval_seconds, 1):
+1 -1
View File
@@ -410,7 +410,7 @@ async def rows():
phases.mark("window") phases.mark("window")
# The routers' metric anomalies, last 30 days (#5741). # The routers' metric anomalies, last 30 days (#5741).
anomalies = await anom.views(db, await anom.list_anomalies( anomalies = await anom.views(db, await anom.list_anomalies(
db, source="traefik", since=now - timedelta(days=30), limit=20), links=False) db, source="traefik", since=now - timedelta(days=30), limit=100), links=False)
for r in current: for r in current:
points = history[r["name"]] points = history[r["name"]]
r["since"], r["since_before"] = (problem_since(points, thresholds) if r["state"] != "ok" r["since"], r["since_before"] = (problem_since(points, thresholds) if r["state"] != "ok"
+2
View File
@@ -17,6 +17,8 @@ dependencies = [
"packaging>=24.0", "packaging>=24.0",
"click>=8.0", "click>=8.0",
"httpx>=0.27", "httpx>=0.27",
# Floor: websockets.asyncio.client first ships in 13.
"websockets>=13",
"cryptography>=42", "cryptography>=42",
# Floor, not a cap: MCPServer (mcp.server.mcpserver) and the mcp.types # Floor, not a cap: MCPServer (mcp.server.mcpserver) and the mcp.types
# mirror that steward/mcp/ is written against first ship in SDK 2.2. # mirror that steward/mcp/ is written against first ship in SDK 2.2.
+13 -4
View File
@@ -284,15 +284,24 @@ async def alert_history():
@require_role(UserRole.viewer) @require_role(UserRole.viewer)
async def anomalies(): async def anomalies():
"""Metric anomalies (#5740): those to review, newest occurrence first, """Metric anomalies (#5740): those to review, newest occurrence first,
then the reviewed and lapsed ones under day dividers.""" then the reviewed and lapsed ones under day dividers. `source` and
`resource` narrow it to one resource's, its parts included: a resource's
page links here with them."""
from urllib.parse import urlencode
from steward.audit.service import mark_day_groups from steward.audit.service import mark_day_groups
from steward.core import anomalies as anom from steward.core import anomalies as anom
from steward.core import formatting as fmt from steward.core import formatting as fmt
source = request.args.get("source", "").strip() or None
resource = request.args.get("resource", "").strip() or None
only = {"source": source, "resource": resource}
async with current_app.db_sessionmaker() as db: async with current_app.db_sessionmaker() as db:
to_review = await anom.views(db, await anom.list_anomalies(db, (anom.TO_REVIEW,))) to_review = await anom.views(db, await anom.list_anomalies(db, (anom.TO_REVIEW,), **only))
past = await anom.views(db, await anom.list_anomalies( past = await anom.views(db, await anom.list_anomalies(
db, (anom.NORMAL, anom.REAL, anom.LAPSED), limit=PAGE_SIZE * 4)) db, (anom.NORMAL, anom.REAL, anom.LAPSED), limit=PAGE_SIZE * 4, **only))
days = int(await get_setting(db, "anomalies.review_days") or 14) days = int(await get_setting(db, "anomalies.review_days") or 14)
here = "/alerts/anomalies"
if source or resource:
here += "?" + urlencode({k: v for k, v in only.items() if v})
tz = fmt.zone(session.get("tz")) tz = fmt.zone(session.get("tz"))
to_review = [{"v": v} for v in to_review] to_review = [{"v": v} for v in to_review]
past = [{"v": v} for v in past] past = [{"v": v} for v in past]
@@ -300,7 +309,7 @@ async def anomalies():
mark_day_groups(to_review, tz, at=at) mark_day_groups(to_review, tz, at=at)
mark_day_groups(past, tz, at=at) mark_day_groups(past, tz, at=at)
return await render_template("alerts/anomalies.html", to_review=to_review, past=past, return await render_template("alerts/anomalies.html", to_review=to_review, past=past,
review_days=days) review_days=days, source=source, resource=resource, here=here)
@alerts_bp.post("/anomalies/<anomaly_id>/review") @alerts_bp.post("/anomalies/<anomaly_id>/review")
+4 -1
View File
@@ -112,7 +112,8 @@ async def resource_url_for(db, source: str, resource: str) -> str | None:
"""Where a (source, resource) of the metric store is shown, or None. """Where a (source, resource) of the metric store is shown, or None.
host_agent resources are the host's name, or "<host>:<mount|nic|core>"; host_agent resources are the host's name, or "<host>:<mount|nic|core>";
docker's are "<host>/<container>"; a monitor's is its name under its type. docker's are "<host>/<container>" or "swarm/<service>"; a monitor's is its
name under its type.
""" """
from steward.models.hosts import Host from steward.models.hosts import Host
if source == "host_agent": if source == "host_agent":
@@ -125,6 +126,8 @@ async def resource_url_for(db, source: str, resource: str) -> str | None:
select(Monitor.id).where(Monitor.type == source, Monitor.name == resource).limit(1) select(Monitor.id).where(Monitor.type == source, Monitor.name == resource).limit(1)
)).scalar_one_or_none() )).scalar_one_or_none()
return f"/monitors/{monitor_id}" if monitor_id else None return f"/monitors/{monitor_id}" if monitor_id else None
if source == "docker" and resource.startswith("swarm/"):
return f"/plugins/docker/service/{resource.split('/', 1)[1]}"
if source == "docker" and "/" in resource: if source == "docker" and "/" in resource:
host_name, container = resource.split("/", 1) host_name, container = resource.split("/", 1)
host_id = (await db.execute( host_id = (await db.execute(
+5 -1
View File
@@ -214,6 +214,10 @@ ACTIONS = {
"prometheus.template_created": "added Prometheus widget template", "prometheus.template_created": "added Prometheus widget template",
"prometheus.template_updated": "changed Prometheus widget template", "prometheus.template_updated": "changed Prometheus widget template",
"prometheus.template_deleted": "deleted Prometheus widget template", "prometheus.template_deleted": "deleted Prometheus widget template",
"homeassistant.connection_saved": "signed Steward in to Home Assistant",
"homeassistant.connection_removed": "removed the Home Assistant connection",
"homeassistant.update_started": "started a Home Assistant update",
"homeassistant.schedule_saved": "set Home Assistant's update schedule",
"syslog.source_added": "added syslog source", "syslog.source_added": "added syslog source",
"syslog.source_updated": "changed syslog source", "syslog.source_updated": "changed syslog source",
"syslog.source_removed": "removed syslog source", "syslog.source_removed": "removed syslog source",
@@ -244,7 +248,7 @@ AREAS = (("auth.", "Sign-ins"), ("account.", "Accounts"), ("user.", "Users"), ("
("ansible.run", "Playbook runs"), ("ansible.", "Ansible (all)"), ("ansible.run", "Playbook runs"), ("ansible.", "Ansible (all)"),
("settings.", "Settings"), ("plugin.", "Plugins"), ("api_token.", "AI access"), ("settings.", "Settings"), ("plugin.", "Plugins"), ("api_token.", "AI access"),
("docker.", "Docker"), ("snmp.", "SNMP"), ("prometheus.", "Prometheus"), ("docker.", "Docker"), ("snmp.", "SNMP"), ("prometheus.", "Prometheus"),
("syslog.", "Syslog"), ("backup.", "Backups"), ("timeline.", "Timeline"), ("syslog.", "Syslog"), ("homeassistant.", "Home Assistant"), ("backup.", "Backups"), ("timeline.", "Timeline"),
("ticket.", "Tickets")) ("ticket.", "Tickets"))
+9 -2
View File
@@ -29,6 +29,8 @@ KEPT: frozenset[str] = frozenset({
# Syslog senders: which were added, under what name, and which ignored. # Syslog senders: which were added, under what name, and which ignored.
"syslog_sources", "syslog_sources",
"syslog_patterns", "syslog_patterns",
"homeassistant_connection",
"homeassistant_schedule",
}) })
# Rows that are measurements, logs, history or in-flight work: left out. # Rows that are measurements, logs, history or in-flight work: left out.
@@ -67,6 +69,7 @@ EXCLUDED: frozenset[str] = frozenset({
"unifi_events", "unifi_fw_rules", "unifi_networks", "unifi_port_forwards", "unifi_events", "unifi_fw_rules", "unifi_networks", "unifi_port_forwards",
"unifi_speedtest_results", "unifi_wan_stats", "unifi_wlan_stats", "unifi_speedtest_results", "unifi_wan_stats", "unifi_wlan_stats",
"syslog_lines", "syslog_lines",
"homeassistant_updates",
}) })
# ── What a check compares (#5823) ──────────────────────────────────────────── # ── What a check compares (#5823) ────────────────────────────────────────────
@@ -89,8 +92,11 @@ ACTIVITY: dict[str, frozenset[str]] = {
"monitors": frozenset({"last_checked_at"}), "monitors": frozenset({"last_checked_at"}),
"ansible_schedules": frozenset({"last_run_at", "last_run_id", "last_error"}), "ansible_schedules": frozenset({"last_run_at", "last_run_id", "last_error"}),
"snmp_devices": frozenset({"last_polled_at", "last_ok_at", "last_error", "missing"}), "snmp_devices": frozenset({"last_polled_at", "last_ok_at", "last_error", "missing"}),
"prometheus_targets": frozenset({"last_scraped_at", "last_ok_at", "last_error", "last_kept", "prometheus_targets": frozenset({"refused", "last_scraped_at", "last_ok_at", "last_error", "last_kept",
"last_dropped", "last_capped"}), "last_dropped", "last_capped"}),
"homeassistant_connection": frozenset({"refused", "last_polled_at", "last_ok_at", "last_error",
"updates", "backup", "health"}),
"homeassistant_schedule": frozenset({"last_slot_at", "stopped"}),
# What the sender last sent; its address, name, state and host are setup. # What the sender last sent; its address, name, state and host are setup.
"syslog_sources": frozenset({"hostname_seen", "first_seen_at", "last_seen_at", "lines_total", "syslog_sources": frozenset({"hostname_seen", "first_seen_at", "last_seen_at", "lines_total",
"last_severity", "last_line"}), "last_severity", "last_line"}),
@@ -121,5 +127,6 @@ AREAS: tuple[tuple[str, tuple[str, ...]], ...] = (
("Ansible", ("ansible_targets", "ansible_groups", "ansible_target_groups", "ansible_schedules")), ("Ansible", ("ansible_targets", "ansible_groups", "ansible_target_groups", "ansible_schedules")),
("Dashboards", ("dashboards", "dashboard_widgets", "dashboard_share_tokens")), ("Dashboards", ("dashboards", "dashboard_widgets", "dashboard_share_tokens")),
("Plugins", ("snmp_devices", "prometheus_targets", "prometheus_widget_templates", ("Plugins", ("snmp_devices", "prometheus_targets", "prometheus_widget_templates",
"unifi_known_clients", "syslog_sources", "syslog_patterns")), "unifi_known_clients", "syslog_sources", "syslog_patterns",
"homeassistant_connection", "homeassistant_schedule")),
) )
+3 -2
View File
@@ -65,9 +65,10 @@ def agent_heartbeat(source_module: str, metric_name: str,
_HEARTBEAT = Heartbeat(source_module, metric_name, offline_after) _HEARTBEAT = Heartbeat(source_module, metric_name, offline_after)
def agent_fed(source_module: str, host_of: Callable[[str], str]) -> None: def agent_fed(source_module: str, host_of: Callable[[str], str | None]) -> None:
"""A source whose samples a host's agent carries; `host_of(resource)` """A source whose samples a host's agent carries; `host_of(resource)`
names the host ("nas:/" → "nas", "nas/web" → "nas").""" names the host ("nas:/" → "nas", "nas/web" → "nas"), or None for a
resource no one host carries."""
_AGENT_FED[source_module] = host_of _AGENT_FED[source_module] = host_of
+61 -23
View File
@@ -4,10 +4,13 @@ Every minute the last WINDOW of each learned key is averaged and compared
with the normal for the slot it falls in (steward.core.norms). A level above with the normal for the slot it falls in (steward.core.norms). A level above
the slot's usual hold by DEPART times, and by at least the metric's floor, has the slot's usual hold by DEPART times, and by at least the metric's floor, has
departed. That opens an occurrence once it is also outside what the normal departed. That opens an occurrence once it is also outside what the normal
records in height or in time: higher than the slot's usual peak, or held, with records in height or in time: higher than the metric usually goes (Norm.
most of the last HELD above the line. A burst of a few minutes within the usual_height), or held, with most of the last HELD above the line. A burst of
usual peak lifts the average for a while and opens nothing. The occurrence ends a few minutes within that lifts the average for a while and opens nothing. The
when the level comes back under the line. height is judged with the window's highest reading left out, so one slow
request or one busy sample can't carry a window past it, and a held level needs
at least HELD_FEWEST readings behind its median: a lone reading never opens
one. The occurrence ends when the level comes back under the line.
What the operator ruled (System 182 "Metric Norms and Anomalies"): What the operator ruled (System 182 "Metric Norms and Anomalies"):
- A departure is a warning, marked as a metric anomaly, and stays "to review" - A departure is a warning, marked as a metric anomaly, and stays "to review"
@@ -32,7 +35,7 @@ from steward.timeline.record import anomaly_changed
WINDOW = timedelta(minutes=15) # the level judged is this window's average WINDOW = timedelta(minutes=15) # the level judged is this window's average
DEPART = 1.25 # times the slot's usual hold that counts as a departure DEPART = 1.25 # times the slot's usual hold that counts as a departure
HELD = timedelta(hours=1) # a level within the usual peak opens once the median of this is out HELD = timedelta(hours=1) # a level within its usual height opens once the median of this is out
# An occurrence that comes back within one window of ending continues the # An occurrence that comes back within one window of ending continues the
# same occurrence: a level wobbling around the line is one event, not ten. # same occurrence: a level wobbling around the line is one event, not ten.
REJOIN = WINDOW REJOIN = WINDOW
@@ -123,30 +126,58 @@ async def levels_now(session, now: datetime) -> dict[tuple[str, str, str], float
return {(s, r, m): float(level) for s, r, m, level in rows} return {(s, r, m): float(level) for s, r, m, level in rows}
# The median, as a burst moves an average of the same stretch but not this. def _per_key(aggregate: str):
_HELD_LEVELS = text(""" """A read of `aggregate` over each given key's readings in [:since, :now].
SELECT k.s, k.r, k.m, v.held A key whose aggregate is NULL is left out."""
return text(f"""
SELECT k.s, k.r, k.m, v.x
FROM unnest(:sources, :resources, :metrics) AS k(s, r, m) FROM unnest(:sources, :resources, :metrics) AS k(s, r, m)
CROSS JOIN LATERAL ( CROSS JOIN LATERAL (
SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) AS held FROM plugin_metrics p SELECT {aggregate} AS x FROM plugin_metrics p
WHERE p.source_module = k.s AND p.resource_name = k.r AND p.metric_name = k.m WHERE p.source_module = k.s AND p.resource_name = k.r AND p.metric_name = k.m
AND p.recorded_at >= :since AND p.recorded_at <= :now AND p.recorded_at >= :since AND p.recorded_at <= :now
) v ) v
WHERE v.held IS NOT NULL WHERE v.x IS NOT NULL
""").bindparams(bindparam("sources", type_=ARRAY(String)), bindparam("resources", type_=ARRAY(String)), """).bindparams(bindparam("sources", type_=ARRAY(String)), bindparam("resources", type_=ARRAY(String)),
bindparam("metrics", type_=ARRAY(String))) bindparam("metrics", type_=ARRAY(String)))
async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]: # The median, as a burst moves an average of the same stretch but not this.
"""{(source, resource, metric): the median of its last HELD}. Asked only of # Fewer than HELD_FEWEST readings in the stretch are not a held level.
keys that have departed and stayed within their usual peak, so the longer HELD_FEWEST = 3
read isn't paid for the rest.""" _HELD_LEVELS = _per_key(
f"CASE WHEN count(*) >= {HELD_FEWEST} THEN percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) END")
# The average with the single highest reading left out; NULL for one reading.
_TOPLESS_LEVELS = _per_key("CASE WHEN count(*) > 1 THEN (sum(p.value) - max(p.value)) / (count(*) - 1) END")
async def _read_per_key(session, query, keys: list[tuple[str, str, str]], since: datetime,
now: datetime) -> dict[tuple[str, str, str], float]:
if not keys: if not keys:
return {} return {}
rows = await session.execute(_HELD_LEVELS, { rows = await session.execute(query, {
"sources": [k[0] for k in keys], "resources": [k[1] for k in keys], "metrics": [k[2] for k in keys], "sources": [k[0] for k in keys], "resources": [k[1] for k in keys], "metrics": [k[2] for k in keys],
"since": now - HELD, "now": now}) "since": since, "now": now})
return {(s, r, m): float(held) for s, r, m, held in rows} return {(s, r, m): float(x) for s, r, m, x in rows}
async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]:
"""{(source, resource, metric): the median of its last HELD}, for keys with
at least HELD_FEWEST readings in it. Asked only of keys that have departed
and not opened by height, so the longer read isn't paid for the rest."""
return await _read_per_key(session, _HELD_LEVELS, keys, now - HELD, now)
async def topless_levels(session, keys: list[tuple[str, str, str]],
now: datetime) -> dict[tuple[str, str, str], float]:
"""{(source, resource, metric): its average over the last WINDOW without
the window's highest reading}. A key with one reading in the window is
left out: one reading is never a level."""
return await _read_per_key(session, _TOPLESS_LEVELS, keys, now - WINDOW, now)
def _past_height(level: float, norm: Norm, floor: float) -> bool:
return level > norm.usual_height and departs(level, norm, floor)
async def detect(session, now: datetime | None = None) -> dict: async def detect(session, now: datetime | None = None) -> dict:
@@ -166,10 +197,17 @@ async def detect(session, now: datetime | None = None) -> dict:
} }
opened = closed = 0 opened = closed = 0
changed: list[MetricAnomaly] = [] # opened, joined or ended: their timeline entries follow changed: list[MetricAnomaly] = [] # opened, joined or ended: their timeline entries follow
out_now = {key for key, level in levels.items() def floor(key):
if key in norms and departs(level, norms[key], LEARNED[(key[0], key[2])].least_departure)} return LEARNED[(key[0], key[2])].least_departure
held = await held_levels(session, [key for key in out_now
if key not in open_now and levels[key] <= norms[key].usual_peak], now) out_now = {key for key, level in levels.items() if key in norms and departs(level, norms[key], floor(key))}
waiting = [key for key in out_now if key not in open_now]
# Past the usual height only if it stays past it without its highest
# reading: one slow request lifts a window's average on its own.
tall = [key for key in waiting if levels[key] > norms[key].usual_height]
topless = await topless_levels(session, tall, now)
past_height = {key for key in tall if key in topless and _past_height(topless[key], norms[key], floor(key))}
held = await held_levels(session, [key for key in waiting if key not in past_height], now)
for key, level in levels.items(): for key, level in levels.items():
norm = norms.get(key) norm = norms.get(key)
out = key in out_now out = key in out_now
@@ -182,10 +220,10 @@ async def detect(session, now: datetime | None = None) -> dict:
_close(anomaly, span, now) _close(anomaly, span, now)
changed.append(anomaly) changed.append(anomaly)
closed += 1 closed += 1
elif out and level > norm.usual_peak: elif key in past_height:
changed.append(await _occur(session, key, level, norm, now, WINDOW)) changed.append(await _occur(session, key, level, norm, now, WINDOW))
opened += 1 opened += 1
elif out and departs(held.get(key, 0.0), norm, LEARNED[(key[0], key[2])].least_departure): elif out and departs(held.get(key, 0.0), norm, floor(key)):
changed.append(await _occur(session, key, level, norm, now, HELD)) changed.append(await _occur(session, key, level, norm, now, HELD))
opened += 1 opened += 1
# An open occurrence whose key stopped reporting, lost its normal or fell # An open occurrence whose key stopped reporting, lost its normal or fell
+3
View File
@@ -298,6 +298,8 @@ def metric_value(name: str, value: Any) -> str:
# its underscores spaced (SNMP's OID labels are the operator's own words). # its underscores spaced (SNMP's OID labels are the operator's own words).
METRICS: dict[str, tuple[str, str]] = { METRICS: dict[str, tuple[str, str]] = {
"is_up": ("Up or down", "whether the check passed"), "is_up": ("Up or down", "whether the check passed"),
"backup_age_hours": ("Age of last backup (hours)", "hours since Home Assistant's last successful automatic backup"),
"updates_waiting": ("Updates waiting", "Core, OS and add-on updates Home Assistant has waiting"),
"is_healthy": ("Health check", "the container's own HEALTHCHECK result"), "is_healthy": ("Health check", "the container's own HEALTHCHECK result"),
"response_ms": ("Response time", "how long the check took"), "response_ms": ("Response time", "how long the check took"),
"ingest_posts": ("Agent posts per minute", "host-agent reports Steward took in the last minute"), "ingest_posts": ("Agent posts per minute", "host-agent reports Steward took in the last minute"),
@@ -421,6 +423,7 @@ SOURCES: dict[str, str] = {
"http": "HTTP monitors", "steward": "Steward itself", "host_agent": "Host agent", "http": "HTTP monitors", "steward": "Steward itself", "host_agent": "Host agent",
"docker": "Docker containers", "traefik": "Traefik routers", "unifi": "UniFi", "docker": "Docker containers", "traefik": "Traefik routers", "unifi": "UniFi",
"snmp": "SNMP devices", "prometheus": "Prometheus targets", "syslog": "Syslog patterns", "snmp": "SNMP devices", "prometheus": "Prometheus targets", "syslog": "Syslog patterns",
"homeassistant": "Home Assistant",
} }
+29 -9
View File
@@ -11,7 +11,12 @@ What a slot's normal is:
(plugin_metrics_hourly.value_p95, #5738), at its 90th percentile across (plugin_metrics_hourly.value_p95, #5738), at its 90th percentile across
past days. A two-minute spike doesn't raise it; an hour spent high does. past days. A two-minute spike doesn't raise it; an hour spent high does.
- usual_peak: the same for each hour's max, so a page can say how high the - usual_peak: the same for each hour's max, so a page can say how high the
slot usually goes. slot usually goes. Most hours of a metric that bursts now and then see no
burst, so this can read 0 for a slot such a metric bursts in as much as any.
- usual_anywhere: how high the metric has gone in most hours of the day, the
median across the 24 hours of each one's highest peak. Bursts that land at
any hour raise it. A job at a time of its own raises a few hours and leaves
the median alone, so the same level at another time still stands out.
The slots: The slots:
- An hour of the day (24 slots) once a key has DAY_MIN_DAYS of history. - An hour of the day (24 slots) once a key has DAY_MIN_DAYS of history.
@@ -116,6 +121,13 @@ class Norm:
usual_hold: float usual_hold: float
usual_peak: float usual_peak: float
days_seen: int days_seen: int
usual_anywhere: float = 0.0
@property
def usual_height(self) -> float:
"""The highest this metric usually goes at this time: a level past it
is news however briefly it lasted."""
return max(self.usual_peak, self.usual_anywhere)
def slot_of(at: datetime, kind: str) -> int: def slot_of(at: datetime, kind: str) -> int:
@@ -171,6 +183,12 @@ kinds AS (
SELECT s, r, m, days, CASE WHEN first <= :week_ready THEN 'week' ELSE 'day' END AS kind SELECT s, r, m, days, CASE WHEN first <= :week_ready THEN 'week' ELSE 'day' END AS kind
FROM keyed WHERE days >= CAST(:day_min_days AS int) FROM keyed WHERE days >= CAST(:day_min_days AS int)
), ),
anywhere AS (
SELECT s, r, m, percentile_cont(0.5) WITHIN GROUP (ORDER BY highest) AS peak
FROM (SELECT s, r, m, max(peak) AS highest FROM hours
GROUP BY s, r, m, EXTRACT(HOUR FROM b AT TIME ZONE 'UTC')) by_hour
GROUP BY s, r, m
),
slotted AS ( slotted AS (
SELECT h.s, h.r, h.m, k.kind, k.days, h.hold, h.peak, SELECT h.s, h.r, h.m, k.kind, k.days, h.hold, h.peak,
CASE k.kind CASE k.kind
@@ -183,13 +201,15 @@ slotted AS (
WHERE k.kind = 'week' OR h.b >= :day_since WHERE k.kind = 'week' OR h.b >= :day_since
) )
INSERT INTO metric_norms (source_module, resource_name, metric_name, slot, kind, INSERT INTO metric_norms (source_module, resource_name, metric_name, slot, kind,
usual_hold, usual_peak, days_seen, updated_at) usual_hold, usual_peak, usual_anywhere, days_seen, updated_at)
SELECT s, r, m, (slot + n.off + slots) % slots, kind, SELECT l.s, l.r, l.m, (l.slot + n.off + l.slots) % l.slots, l.kind,
percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY hold), percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY l.hold),
percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY peak), percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY l.peak),
days, :now a.peak, l.days, :now
FROM slotted CROSS JOIN generate_series(-CAST(:neighbours AS int), CAST(:neighbours AS int)) AS n(off) FROM slotted l
GROUP BY s, r, m, (slot + n.off + slots) % slots, kind, days JOIN anywhere a ON a.s = l.s AND a.r = l.r AND a.m = l.m
CROSS JOIN generate_series(-CAST(:neighbours AS int), CAST(:neighbours AS int)) AS n(off)
GROUP BY l.s, l.r, l.m, (l.slot + n.off + l.slots) % l.slots, l.kind, l.days, a.peak
""").bindparams(bindparam("keys", type_=ARRAY(String)), """).bindparams(bindparam("keys", type_=ARRAY(String)),
bindparam("starts", type_=ARRAY(DateTime(timezone=True))), bindparam("starts", type_=ARRAY(DateTime(timezone=True))),
bindparam("parts", type_=ARRAY(Boolean))) bindparam("parts", type_=ARRAY(Boolean)))
@@ -235,7 +255,7 @@ async def norms_at(session, keys: list[tuple[str, str, str]],
| ((MetricNorm.kind == "week") & (MetricNorm.slot == week))) | ((MetricNorm.kind == "week") & (MetricNorm.slot == week)))
)).scalars().all() )).scalars().all()
return {(n.source_module, n.resource_name, n.metric_name): return {(n.source_module, n.resource_name, n.metric_name):
Norm(n.kind, n.slot, n.usual_hold, n.usual_peak, n.days_seen) for n in rows} Norm(n.kind, n.slot, n.usual_hold, n.usual_peak, n.days_seen, n.usual_anywhere) for n in rows}
async def run_norms(app) -> None: async def run_norms(app) -> None:
+1
View File
@@ -181,6 +181,7 @@ DEFAULTS: dict[str, Any] = {
"plugin.snmp": {"enabled": True}, "plugin.snmp": {"enabled": True},
"plugin.prometheus": {"enabled": True}, "plugin.prometheus": {"enabled": True},
"plugin.syslog": {"enabled": True}, "plugin.syslog": {"enabled": True},
"plugin.homeassistant": {"enabled": True},
# OIDC single-sign-on # OIDC single-sign-on
"oidc.enabled": False, "oidc.enabled": False,
"oidc.discovery_url": "", "oidc.discovery_url": "",
+64
View File
@@ -0,0 +1,64 @@
# steward/core/weekly.py
"""A weekly slot: a weekday and a time in a time zone, with a window after it. A
slot fires once, and only while its window is open, so a Steward that was down
over the slot starts late rather than a day late. A schedule is anything with
weekday, at_time, time_zone, window_minutes, enabled, last_slot_at and created_at."""
from __future__ import annotations
from datetime import datetime, time, timedelta, timezone
from steward.core.formatting import zone
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
def parse_at(at_time: str) -> time:
h, m = (int(x) for x in at_time.split(":", 1))
if not (0 <= h < 24 and 0 <= m < 60):
raise ValueError(at_time)
return time(h, m)
def last_slot(plan, now: datetime) -> datetime:
"""The most recent scheduled time at or before `now`, in UTC."""
tz = zone(plan.time_zone)
local = now.astimezone(tz)
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
if slot > local:
slot -= timedelta(days=7)
return slot.astimezone(timezone.utc)
def fired_slot(plan) -> datetime | None:
"""The last slot that fired, or None while none has. A new schedule's
last_slot_at holds the moment it was made, so its first slot is the
next one rather than one already past; that moment is no slot."""
at = plan.last_slot_at
if at is None:
return None
return at if last_slot(plan, at) == at else None
def next_slot(plan, now: datetime) -> datetime:
return last_slot(plan, now) + timedelta(days=7)
def is_due(plan, now: datetime) -> datetime | None:
"""The slot to fire now, or None."""
if not plan.enabled:
return None
slot = last_slot(plan, now)
since = plan.last_slot_at or plan.created_at
if since is not None and slot <= since:
return None
if now - slot >= timedelta(minutes=plan.window_minutes):
return None
return slot
def describe(plan) -> str:
""""Sundays at 03:00 (Europe/London), up to 4 h" for a page."""
hours = plan.window_minutes / 60
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
+3 -2
View File
@@ -132,10 +132,11 @@ async def host_detail(host_id: str):
linkable_targets = (await db.execute( linkable_targets = (await db.execute(
select(AnsibleTarget).where(AnsibleTarget.host_id.is_(None)) select(AnsibleTarget).where(AnsibleTarget.host_id.is_(None))
.order_by(AnsibleTarget.name))).scalars().all() .order_by(AnsibleTarget.name))).scalars().all()
# The host's metric anomalies, its disks and interfaces included (#5740). # The host's metric anomalies, its disks and interfaces included
# (#5740). The card counts those to review, so the limit is generous.
anomalies = await anom.views(db, await anom.list_anomalies( anomalies = await anom.views(db, await anom.list_anomalies(
db, source="host_agent", resource=host.name, db, source="host_agent", resource=host.name,
since=datetime.now(timezone.utc) - timedelta(days=30), limit=10), links=False) since=datetime.now(timezone.utc) - timedelta(days=30), limit=100), links=False)
return await render_template( return await render_template(
"hosts/detail.html", "hosts/detail.html",
+3 -3
View File
@@ -13,10 +13,10 @@ each argument's empty value means, and what comes back.
from __future__ import annotations from __future__ import annotations
from . import (alerts, anomalies, audit, automation, backups, hosts, metrics, monitors, overview, session, from . import (alerts, anomalies, audit, automation, backups, hosts, metrics, monitors, overview, session,
tickets, timeline) thresholds, tickets, timeline)
_MODULES = (session, overview, hosts, monitors, alerts, anomalies, metrics, audit, timeline, tickets, _MODULES = (session, overview, hosts, monitors, alerts, anomalies, thresholds, metrics, audit, timeline,
automation, backups) tickets, automation, backups)
def register_all(register_tool) -> None: def register_all(register_tool) -> None:
+7 -3
View File
@@ -53,7 +53,7 @@ async def list_anomalies(review: str = "to_review", limit: int = 50) -> dict:
"""Metric anomalies: a learned metric whose 15-minute level went well """Metric anomalies: a learned metric whose 15-minute level went well
above what is normal for it at that hour (its usual hold ×1.25, and by above what is normal for it at that hour (its usual hold ×1.25, and by
more than the metric's floor), and either past how high it usually goes more than the metric's floor), and either past how high it usually goes
(`usual_peak`) or held there for most of an hour. (see get_metric_normal) or held there for most of an hour.
review: "to_review" (default) — waiting for someone to mark them; "normal" review: "to_review" (default) — waiting for someone to mark them; "normal"
— accepted, their hours are learned; "real" — a real problem, not learned; — accepted, their hours are learned; "real" — a real problem, not learned;
@@ -291,8 +291,11 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
90th percentile across past days), `usual_peak` (how high they usually 90th percentile across past days), `usual_peak` (how high they usually
went), `days_seen`, and `flag_above`: the 15-minute average that has went), `days_seen`, and `flag_above`: the 15-minute average that has
departed there (usual hold ×1.25, and at least `floor` above it). A departed there (usual hold ×1.25, and at least `floor` above it). A
departure opens an anomaly once it passes `usual_peak`, or when the median departure opens an anomaly once it passes both `usual_peak` and
of the last hour is over `flag_above` too. `usual_anywhere` (how high the metric has gone in most hours of the day,
one figure for the whole key) even without the window's highest reading,
or when the median of the last hour (three readings or more) is over
`flag_above` too. A lone slow reading opens nothing.
`floor` is the smallest departure worth calling an anomaly, and `floor` is the smallest departure worth calling an anomaly, and
`floor_setting` the Settings → Thresholds floor it comes from. Hours an `floor_setting` the Settings → Thresholds floor it comes from. Hours an
anomaly covers are learned only once someone marks it normal. anomaly covers are learned only once someone marks it normal.
@@ -319,6 +322,7 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
out = { out = {
"source_module": source_module, "resource": resource, "metric": metric, "source_module": source_module, "resource": resource, "metric": metric,
"learned": bool(slots), "kind": kind, "learned": bool(slots), "kind": kind,
"usual_anywhere": rnd(rows[0].usual_anywhere, 3) if rows else None,
"floor": rnd(floor, 3), "floor_setting": learned.floor_name, "floor": rnd(floor, 3), "floor_setting": learned.floor_name,
"only_while": ({"metric": learned.gate[0], "only_while": ({"metric": learned.gate[0],
"at_least": rnd(learned.gate[1]() if callable(learned.gate[1]) else learned.gate[1], 3), "at_least": rnd(learned.gate[1]() if callable(learned.gate[1]) else learned.gate[1], 3),
+24 -2
View File
@@ -1,6 +1,8 @@
"""Host tools: the fleet, one host in depth, and a host's agent metrics.""" """Host tools: the fleet, one host in depth, and a host's agent metrics."""
from __future__ import annotations from __future__ import annotations
from typing import Callable
from sqlalchemy import select from sqlalchemy import select
from sqlalchemy.orm import selectinload from sqlalchemy.orm import selectinload
@@ -19,6 +21,18 @@ from .tickets import open_tickets_for
AGENT_SOURCE = "host_agent" AGENT_SOURCE = "host_agent"
# Which of a host's sensors don't count as its temperature, and why: unwired
# inputs read a constant or move against the CPU. The plugin that knows its
# sensors registers the judge; with none, every sensor counts.
_UNCOUNTED_SENSORS: Callable[[str, dict[str, float]], dict[str, str]] | None = None
def judge_sensors(judge: Callable[[str, dict[str, float]], dict[str, str]]) -> None:
"""`judge(host_name, {sensor: celsius})` → {sensor: why} for each sensor
that is not a thermometer; get_host marks those `counted: false`."""
global _UNCOUNTED_SENSORS
_UNCOUNTED_SENSORS = judge
async def _all_hosts(db) -> list[Host]: async def _all_hosts(db) -> list[Host]:
return list((await db.execute(select(Host).order_by(Host.name))).scalars()) return list((await db.execute(select(Host).order_by(Host.name))).scalars())
@@ -50,7 +64,7 @@ _PART_ORDER = {
"interfaces": lambda r: -((r.get("net_rx_bps") or 0) + (r.get("net_tx_bps") or 0)), "interfaces": lambda r: -((r.get("net_rx_bps") or 0) + (r.get("net_tx_bps") or 0)),
"disk_io": lambda r: -((r.get("disk_read_bps") or 0) + (r.get("disk_write_bps") or 0)), "disk_io": lambda r: -((r.get("disk_read_bps") or 0) + (r.get("disk_write_bps") or 0)),
"drive_health": lambda r: (-(r.get("drive_failing") or 0), -(r.get("drive_bad_sectors") or 0)), "drive_health": lambda r: (-(r.get("drive_failing") or 0), -(r.get("drive_bad_sectors") or 0)),
"temperatures": lambda r: -(r.get("temp_c") or 0), "temperatures": lambda r: (not r.get("counted", True), -(r.get("temp_c") or 0)),
"other": lambda r: r["part"], "other": lambda r: r["part"],
} }
@@ -79,6 +93,12 @@ def agent_parts(parts: dict[str, dict], host_name: str, limit: int) -> dict:
groups["mounts"].append({"mount": name, "part": name, **metrics}) groups["mounts"].append({"mount": name, "part": name, **metrics})
else: else:
groups["other"].append({"part": name, **metrics}) groups["other"].append({"part": name, **metrics})
temps = {r["sensor"]: r["temp_c"] for r in groups["temperatures"] if r.get("temp_c") is not None}
uncounted = _UNCOUNTED_SENSORS(host_name, temps) if _UNCOUNTED_SENSORS and temps else {}
for r in groups["temperatures"]:
r["counted"] = r["sensor"] not in uncounted
if not r["counted"]:
r["not_counted_because"] = uncounted[r["sensor"]]
more = (f"Pass a larger `parts_limit` (max 200), or read one part with query_metrics(" more = (f"Pass a larger `parts_limit` (max 200), or read one part with query_metrics("
f'source_module="host_agent", resource="{host_name}:<part>").') f'source_module="host_agent", resource="{host_name}:<part>").')
out: dict = {} out: dict = {}
@@ -148,7 +168,9 @@ async def get_host(host: str, parts_limit: int = 20) -> dict:
The agent's values are `agent.metrics` (host-level) and its sub-resources The agent's values are `agent.metrics` (host-level) and its sub-resources
grouped by kind: `mounts` (fullest first), `interfaces` and `disk_io` grouped by kind: `mounts` (fullest first), `interfaces` and `disk_io`
(busiest first), `temperatures` (hottest first), each with `items`, (busiest first), `temperatures` (the ones that count as the host's
temperature hottest first, then those left out of it: `counted: false`
with `not_counted_because`, likely nothing wired to them), each with `items`,
`total` and `truncated`; `cpu_cores` summarises per-core CPU (count, `total` and `truncated`; `cpu_cores` summarises per-core CPU (count,
min/avg/max, the three busiest). One part's history is query_metrics with min/avg/max, the three busiest). One part's history is query_metrics with
source_module="host_agent" and resource="<host>:<part>", where `part` is source_module="host_agent" and resource="<host>:<part>", where `part` is
+94
View File
@@ -0,0 +1,94 @@
"""The cutoffs a host's readings are graded by, as Settings → Thresholds sets
them: amber past `warn`, red past `crit`."""
from __future__ import annotations
from steward.core.settings import DEFAULTS, get_all_settings, set_setting, to_thresholds_cfg
from steward.mcp._context import mcp_app
from steward.models.users import UserRole
from ._audit import audit_mcp
from ._shape import db_session
# Each graded reading, its unit, and what it measures.
READINGS = {
"cpu": ("%", "CPU use"),
"mem": ("%", "memory used"),
"disk": ("%", "fullest mount's space used"),
"load": ("%", "1-minute load per core"),
"temp": ("°C", "hottest real thermometer"),
}
def _view(settings: dict) -> dict:
cfg = to_thresholds_cfg(settings)
return {kind: {"warn": cfg[kind]["warn"], "crit": cfg[kind]["crit"], "unit": unit, "measures": what,
"default": {"warn": DEFAULTS[f"thresholds.{kind}_warn"],
"crit": DEFAULTS[f"thresholds.{kind}_crit"]}}
for kind, (unit, what) in READINGS.items()}
async def get_thresholds() -> dict:
"""The cutoffs host readings are graded by, as Settings → Thresholds
sets them: per reading (cpu, mem, disk, load, temp), `warn` (amber past
it) and `crit` (red past it) in `unit`, with Steward's default for each.
The host pages, the Overview and get_overview grade by these. Alert
rules have their own cutoffs (list_alert_rules); these don't fire
anything. set_thresholds (admin) changes them.
"""
async with db_session() as db:
return _view(await get_all_settings(db))
async def set_thresholds(cutoffs: dict[str, dict[str, float]]) -> dict:
"""Change the cutoffs host readings are graded by, as Settings →
Thresholds does. Takes effect on the next page load; audited.
cutoffs: {reading: {"warn": n, "crit": n}} for any of get_thresholds'
readings, e.g. {"temp": {"crit": 90}}. Either of warn and crit may be
left out and keeps its value; warn must stay below crit. A reading not
named is left as it is. Returns the cutoffs as they now stand.
"""
unknown = sorted(set(cutoffs or {}) - set(READINGS))
if unknown:
raise ValueError(f"No reading named {', '.join(unknown)}; they are {', '.join(READINGS)}.")
if not cutoffs:
raise ValueError("Nothing to change: pass cutoffs, e.g. {\"temp\": {\"crit\": 90}}.")
async with db_session() as db:
current = _view(await get_all_settings(db))
changes: dict = {}
for kind, given in cutoffs.items():
extra = sorted(set(given or {}) - {"warn", "crit"})
if extra or not given:
raise ValueError(f"{kind} takes warn and/or crit, not {', '.join(extra) or 'nothing'}.")
new = {}
for edge in ("warn", "crit"):
if edge not in given:
new[edge] = current[kind][edge]
continue
try:
value = float(given[edge])
except (TypeError, ValueError):
raise ValueError(f"{kind} {edge} must be a number.") from None
if not value > 0:
raise ValueError(f"{kind} {edge} must be more than 0.")
new[edge] = int(value) if value.is_integer() else value
changes[f"thresholds.{kind}_{edge}"] = new[edge]
if new["warn"] >= new["crit"]:
raise ValueError(f"{kind} warn ({new['warn']}) must be below its crit ({new['crit']}).")
app = mcp_app()
async with db_session() as db:
async with db.begin():
before = await get_all_settings(db)
for key, value in changes.items():
await set_setting(db, key, value)
fresh = await get_all_settings(db)
app.config["THRESHOLDS"] = to_thresholds_cfg(fresh)
await audit_mcp("settings.saved", "settings", "thresholds",
{"section": "thresholds",
"changes": {k: {"from": before.get(k), "to": v} for k, v in changes.items()}})
return _view(fresh)
def register(register_tool) -> None:
register_tool(get_thresholds, role=UserRole.viewer, read_only=True)
register_tool(set_thresholds, role=UserRole.admin, read_only=False)
@@ -0,0 +1,23 @@
"""Metric normals: how high a metric has gone in most hours of the day.
Revision ID: 0050_norm_usual_anywhere
Revises: 0049_alert_on_a_rise
Create Date: 2026-10-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0050_norm_usual_anywhere"
down_revision: Union[str, None] = "0049_alert_on_a_rise"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("metric_norms", sa.Column("usual_anywhere", sa.Float, nullable=False, server_default="0"))
def downgrade() -> None:
op.drop_column("metric_norms", "usual_anywhere")
+2
View File
@@ -24,6 +24,8 @@ class MetricNorm(Base):
# peaked: the 90th percentile, across past days, of each hour's p95 and max. # peaked: the 90th percentile, across past days, of each hour's p95 and max.
usual_hold: Mapped[float] = mapped_column(Float, nullable=False) usual_hold: Mapped[float] = mapped_column(Float, nullable=False)
usual_peak: Mapped[float] = mapped_column(Float, nullable=False) usual_peak: Mapped[float] = mapped_column(Float, nullable=False)
# How high the metric has gone in most hours of the day, whatever the slot.
usual_anywhere: Mapped[float] = mapped_column(Float, nullable=False, default=0.0, server_default="0")
days_seen: Mapped[int] = mapped_column(Integer, nullable=False) days_seen: Mapped[int] = mapped_column(Integer, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+1 -1
View File
@@ -128,7 +128,7 @@ async def detail(monitor_id: str):
norm = (await monitor_svc.latency_norms(db, [m])).get(m.id) norm = (await monitor_svc.latency_norms(db, [m])).get(m.id)
anomalies = await anom.views(db, await anom.list_anomalies( anomalies = await anom.views(db, await anom.list_anomalies(
db, source=m.type, resource=m.name, db, source=m.type, resource=m.name,
since=datetime.now(timezone.utc) - timedelta(days=30), limit=20), links=False) since=datetime.now(timezone.utc) - timedelta(days=30), limit=100), links=False)
host = await db.get(Host, m.host_id) if m.host_id else None host = await db.get(Host, m.host_id) if m.host_id else None
thresholds = current_app.config.get("THRESHOLDS", {}) thresholds = current_app.config.get("THRESHOLDS", {})
latest = st["latest"] latest = st["latest"]
+3 -1
View File
@@ -446,7 +446,9 @@ a.stat-tile:hover { border-color: var(--border-mid); background: var(--bg-elevat
.host-health-summary { font-size: 0.92rem; } .host-health-summary { font-size: 0.92rem; }
/* Its four tiles share the card's width (two a row on a phone); each sparkline /* Its four tiles share the card's width (two a row on a phone); each sparkline
stretches across its tile. */ stretches across its tile. */
.host-health-tiles { grid-template-columns: repeat(4, minmax(0, 1fr)); } .host-health-tiles { grid-template-columns: repeat(var(--tiles, 4), minmax(0, 1fr)); }
/* Which sensor the temperature tile reads, under its line; cut with … when long. */
.stat-tile-sensor { display: block; font-size: 0.72rem; color: var(--text-muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.host-health-tiles .stat-tile { min-width: 0; } .host-health-tiles .stat-tile { min-width: 0; }
.host-health-tiles .stat-tile-hint { margin-top: 0.25rem; } .host-health-tiles .stat-tile-hint { margin-top: 0.25rem; }
@media (max-width: 640px) { .host-health-tiles { grid-template-columns: repeat(2, minmax(0, 1fr)); } } @media (max-width: 640px) { .host-health-tiles { grid-template-columns: repeat(2, minmax(0, 1fr)); } }
+27 -10
View File
@@ -1,14 +1,31 @@
{# A resource's metric anomalies (#5740), for its own page: `anomalies` are {# A resource's metric anomalies, for its own page: one line saying how many
anomalies.views() rows, to review first; `back` is this page's path, so a wait for review and which are out of their normal now, with a button to
review returns here. Shown only when there is one. #} Alerts → Anomalies narrowed to this resource, where they are reviewed.
{% from "_anomaly.html" import anomaly_row with context %} `anomalies` are anomalies.views() rows; `base_resource` is the page's
resource (none for a page covering a whole source, like Traefik). Shown
only while one waits for review or is going on. #}
{% if anomalies %} {% if anomalies %}
{% set going = anomalies | selectattr("ongoing") | list %} {% set waiting = anomalies | selectattr("anomaly.review", "equalto", "to_review") | list %}
<section class="card{% if going %} problem-card{% endif %}" aria-labelledby="resource-anomalies" style="margin-bottom:1rem;"> {% set going = anomalies | selectattr("ongoing") | rejectattr("anomaly.review", "equalto", "normal") | list %}
<h2 id="resource-anomalies" class="section-title" style="margin-bottom:0.35rem;"> {% if waiting or going %}
{% if going %}Out of its normal now{% else %}Metric anomalies, last 30 days{% endif %} {% set q = {"source": anomalies[0].anomaly.source_module} %}
{% if base_resource %}{% set _ = q.update({"resource": base_resource}) %}{% endif %}
<section class="card{% if going %} problem-card{% endif %}" aria-labelledby="resource-anomalies"
style="margin-bottom:1rem;display:flex;align-items:center;gap:0.5rem 1rem;flex-wrap:wrap;">
<h2 id="resource-anomalies" class="section-title" style="margin:0;">
{% if going %}Out of its normal now{% else %}Metric anomalies{% endif %}
</h2> </h2>
{% for v in anomalies %}{{ anomaly_row(v, back=back, show_resource=v.anomaly.resource_name != base_resource) }}{% endfor %} <span style="font-size:0.85rem;">
<div style="font-size:0.78rem;margin-top:0.4rem;"><a href="/alerts/anomalies">All anomalies</a></div> {%- if going -%}
{%- for v in going -%}
{%- if v.anomaly.resource_name != base_resource %}{{ v.anomaly.resource_name }} {% endif -%}
{{ metric_label(v.anomaly.metric_name) }}{{ ", " if not loop.last }}
{%- endfor -%}
{%- if waiting %} · {% endif -%}
{%- endif -%}
{%- if waiting %}<strong>{{ waiting|length }}</strong> to review{% endif -%}
</span>
<a class="btn btn-sm" style="margin-left:auto;" href="/alerts/anomalies?{{ q|urlencode }}">Review</a>
</section> </section>
{% endif %} {% endif %}
{% endif %}
+8 -1
View File
@@ -11,10 +11,17 @@
{% macro divided(items) %} {% macro divided(items) %}
{% for item in items %} {% for item in items %}
{% if item.opens_group %}<h3 class="table-divider" style="font-size:0.78rem;font-weight:500;color:var(--text);background:var(--bg-elevated);padding:0.35rem 0.75rem;margin:0.75rem -0.75rem 0;border-top:1px solid var(--border-mid);border-bottom:1px solid var(--border-mid);">{{ item.group }}</h3>{% endif %} {% if item.opens_group %}<h3 class="table-divider" style="font-size:0.78rem;font-weight:500;color:var(--text);background:var(--bg-elevated);padding:0.35rem 0.75rem;margin:0.75rem -0.75rem 0;border-top:1px solid var(--border-mid);border-bottom:1px solid var(--border-mid);">{{ item.group }}</h3>{% endif %}
{{ anomaly_row(item.v, back="/alerts/anomalies") }} {{ anomaly_row(item.v, back=here|default("/alerts/anomalies"), show_resource=not resource or item.v.anomaly.resource_name != resource) }}
{% endfor %} {% endfor %}
{% endmacro %} {% endmacro %}
{% if source or resource %}
<p class="field-hint" style="margin:0.75rem 0 0;">
Only {% if resource %}<strong>{{ resource }}</strong>{% if source %} ({{ source_label(source) }}){% endif %}{% else %}{{ source_label(source) }}{% endif %}.
<a href="/alerts/anomalies">Show every anomaly</a>
</p>
{% endif %}
<div class="card" style="margin-top:1rem;"> <div class="card" style="margin-top:1rem;">
<h2 class="section-title" style="margin-bottom:0.35rem;"> <h2 class="section-title" style="margin-bottom:0.35rem;">
{% if to_review %}{{ to_review|length }} to review{% else %}Nothing to review{% endif %} {% if to_review %}{{ to_review|length }} to review{% else %}Nothing to review{% endif %}
+1 -1
View File
@@ -61,7 +61,7 @@
<div style="display:flex;align-items:baseline;justify-content:space-between;margin-bottom:0.5rem;"> <div style="display:flex;align-items:baseline;justify-content:space-between;margin-bottom:0.5rem;">
<h3 class="section-title" style="margin-bottom:0;">Monitors</h3> <h3 class="section-title" style="margin-bottom:0;">Monitors</h3>
{% if session.user_role in ['operator', 'admin'] %} {% if session.user_role in ['operator', 'admin'] %}
<a href="/monitors/new?host_id={{ host.id }}" class="btn btn-ghost btn-sm" style="margin-left:auto;margin-right:0.75rem;">Add monitor</a> <a href="/monitors/new?host_id={{ host.id }}" class="btn btn-ghost btn-sm" style="margin-left:auto;margin-right:0.75rem;flex-shrink:0;" title="Add a monitor for this host">Add</a>
{% endif %} {% endif %}
{% if uptime %} {% if uptime %}
<span style="font-size:0.8rem;color:var(--text-muted);"> <span style="font-size:0.8rem;color:var(--text-muted);">
+3 -1
View File
@@ -118,7 +118,7 @@ for _t in (
# A source not listed here is Monitoring. # A source not listed here is Monitoring.
_SOURCE_AREAS = { _SOURCE_AREAS = {
"icmp": "network", "dns": "network", "tcp": "service", "http": "service", "icmp": "network", "dns": "network", "tcp": "service", "http": "service",
"docker": "service", "prometheus": "service", "docker": "service", "prometheus": "service", "homeassistant": "service",
"traefik": "network", "unifi": "network", "traefik": "network", "unifi": "network",
"snmp": "hardware", "snmp": "hardware",
} }
@@ -154,6 +154,8 @@ _AUDIT_TYPES = (
("account.time_zone", None), ("account.email", None), ("account.time_zone", None), ("account.email", None),
("host_agent.maintenance_plan", "steward.config"), ("host_agent.maintenance", None), ("host_agent.maintenance_plan", "steward.config"), ("host_agent.maintenance", None),
("docker.prune.", None), ("docker.prune.", None),
# An update is on the timeline as itself, once it ends (homeassistant.timeline).
("homeassistant.update", None),
# A change recorded by hand is on the timeline as itself (timeline.changes). # A change recorded by hand is on the timeline as itself (timeline.changes).
("timeline.change", None), ("timeline.change", None),
# A ticket's opening, resolution and reopening are on the timeline as # A ticket's opening, resolution and reopening are on the timeline as
+2 -2
View File
@@ -73,12 +73,12 @@ _MONITOR_SOURCES = ("icmp", "tcp", "dns", "http")
async def hosts_of_rule(session, source: str, resource: str) -> list[str]: async def hosts_of_rule(session, source: str, resource: str) -> list[str]:
"""The host an alert rule's resource is on, by name, when it has one: an """The host an alert rule's resource is on, by name, when it has one: an
agent series ("<host>" or "<host>:<part>"), a container ("<host>/<name>") agent series ("<host>" or "<host>:<part>"), a container ("<host>/<name>")
or a monitor tied to a host.""" or a monitor tied to a host. A swarm service ("swarm/<service>") has none."""
from steward.core.alerts import host_of_resource from steward.core.alerts import host_of_resource
if source == "host_agent": if source == "host_agent":
return [host_of_resource(source, resource) or resource.split(":", 1)[0]] return [host_of_resource(source, resource) or resource.split(":", 1)[0]]
if source == "docker": if source == "docker":
return [resource.split("/", 1)[0]] return [] if resource.startswith("swarm/") else [resource.split("/", 1)[0]]
if source in _MONITOR_SOURCES: if source in _MONITOR_SOURCES:
from steward.models.hosts import Host from steward.models.hosts import Host
from steward.models.monitors import Monitor from steward.models.monitors import Monitor
+39
View File
@@ -88,6 +88,45 @@ async def test_a_quiet_page_says_nothing_waits(app):
assert "Nothing to review" in page and "within 14 days" in page assert "Nothing to review" in page and "within 14 days" in page
async def test_a_page_narrowed_to_one_resource_says_so_and_a_review_returns_to_it(app):
here = "/alerts/anomalies?source=host_agent&resource=nas"
async with app.test_request_context(here):
session["user_id"], session["user_role"] = "u1", "operator"
page = await render_template("alerts/anomalies.html", review_days=14, past=[],
to_review=[{"v": _view(), "opens_group": True, "group": "Today"}],
source="host_agent", resource="nas", here=here)
assert "<strong>nas</strong>" in page and '<a href="/alerts/anomalies">Show every anomaly</a>' in page
assert 'name="next" value="/alerts/anomalies?source=host_agent&amp;resource=nas"' in page
async def _card(app, anomalies, base_resource="nas"):
async with app.test_request_context("/hosts/h1"):
session["user_id"], session["user_role"] = "u1", "operator"
return await render_template("_anomaly_card.html", anomalies=anomalies,
back="/hosts/h1", base_resource=base_resource)
async def test_a_resources_card_is_one_line_counting_what_waits_with_a_way_to_review(app):
ended = NOW - timedelta(minutes=5)
page = await _card(app, [_view(ended_at=ended), _view(id="a2", metric_name="load_pct", ended_at=ended),
_view(id="a3", review="normal", ended_at=ended)])
assert "<strong>2</strong> to review" in page
assert 'href="/alerts/anomalies?source=host_agent&amp;resource=nas"' in page
assert "This is normal" not in page and "Out of its normal now" not in page # reviewed there
async def test_a_card_names_what_is_out_of_its_normal_now(app):
page = await _card(app, [_view(resource_name="nas:/var", metric_name="disk_used_pct")])
assert "Out of its normal now" in page and "nas:/var" in page and "problem-card" in page
async def test_a_card_with_nothing_waiting_or_going_is_not_shown(app):
reviewed = _view(review="normal", ended_at=NOW - timedelta(minutes=5))
assert "resource-anomalies" not in await _card(app, [reviewed])
going_but_accepted = _view(review="normal") # the new normal being learned
assert "resource-anomalies" not in await _card(app, [going_but_accepted])
async def _widget(app, logged_in, anomalies, to_review): async def _widget(app, logged_in, anomalies, to_review):
async with app.test_request_context("/status/widget"): async with app.test_request_context("/status/widget"):
if logged_in: if logged_in:
+3 -1
View File
@@ -42,6 +42,8 @@ async def test_ansible_comes_before_the_docker_list(app, monkeypatch):
# Monitors are added on their own form, filed under this host (#5277); # Monitors are added on their own form, filed under this host (#5277);
# each row's own page has the labelled Edit/Pause/Delete. # each row's own page has the labelled Edit/Pause/Delete.
assert 'href="/monitors/new?host_id=h1"' in page assert 'href="/monitors/new?host_id=h1"' in page
# The card's header shares one line with the uptime: a one-word button keeps it whole.
assert 'title="Add a monitor for this host">Add</a>' in page
assert "row-actions" not in page and 'action="/monitors/add"' not in page assert "row-actions" not in page and 'action="/monitors/add"' not in page
@@ -70,7 +72,7 @@ async def test_the_agent_card_keeps_its_actions_under_manage(app):
app, "host_agent", "panel.html", host=host, reg=reg, reporting=True, stale=False, app, "host_agent", "panel.html", host=host, reg=reg, reporting=True, stale=False,
target=target, ansible_available=True, managed_key_set=True) target=target, ansible_available=True, managed_key_set=True)
manage = page.index(">Manage agent</summary>") manage = page.index(">Manage agent</summary>")
assert page.index("Full metrics →") < manage assert "Full metrics" not in page # it heads the vitals strip, at the top of the page
for action in (">Update agent</button>", ">Rotate token</button>", ">Unregister agent</button>"): for action in (">Update agent</button>", ">Rotate token</button>", ">Unregister agent</button>"):
assert page.index(action) > manage assert page.index(action) > manage
assert ">Remove</button>" not in page assert ">Remove</button>" not in page
+1 -1
View File
@@ -9,7 +9,7 @@ import steward
from steward.alerts.service import CORE_METRICS from steward.alerts.service import CORE_METRICS
_REPO = pathlib.Path(steward.__file__).parent.parent _REPO = pathlib.Path(steward.__file__).parent.parent
_PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp", "prometheus") _PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp", "prometheus", "homeassistant")
def catalog() -> dict[str, set[str]]: def catalog() -> dict[str, set[str]]:
+96 -5
View File
@@ -41,11 +41,13 @@ def src():
norms.LEARNED.pop((name, METRIC), None) norms.LEARNED.pop((name, METRIC), None)
def _norms(src: str, res: str, hold: float = 10.0, peak: float = 20.0): def _norms(src: str, res: str, hold: float = 10.0, peak: float = 20.0, anywhere: float = 0.0):
"""A learned day normal for every hour: holds `hold`, peaks at `peak`.""" """A learned day normal for every hour: holds `hold`, peaks at `peak`, and
reaches `anywhere` in most hours of the day."""
from steward.models.norms import MetricNorm from steward.models.norms import MetricNorm
return [MetricNorm(source_module=src, resource_name=res, metric_name=METRIC, slot=h, return [MetricNorm(source_module=src, resource_name=res, metric_name=METRIC, slot=h,
kind="day", usual_hold=hold, usual_peak=peak, days_seen=7, updated_at=NOW) kind="day", usual_hold=hold, usual_peak=peak, usual_anywhere=anywhere,
days_seen=7, updated_at=NOW)
for h in range(24)] for h in range(24)]
@@ -104,7 +106,8 @@ def test_a_level_within_its_normal_is_no_anomaly(app, src):
assert rows == [] assert rows == []
def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: float = 10.0): def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: float = 10.0,
anywhere: float = 0.0):
"""A detection pass after an hour reading `then`, with its last """A detection pass after an hour reading `then`, with its last
`for_minutes` at `level`, against the normal that holds 10 and peaks at 20.""" `for_minutes` at `level`, against the normal that holds 10 and peaks at 20."""
from sqlalchemy import select from sqlalchemy import select
@@ -115,7 +118,7 @@ def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: f
async def _go(): async def _go():
async with app.db_sessionmaker() as s: async with app.db_sessionmaker() as s:
async with s.begin(): async with s.begin():
s.add_all(_norms(src, res)) s.add_all(_norms(src, res, anywhere=anywhere))
s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC, s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC,
value=level if m < for_minutes else then, value=level if m < for_minutes else then,
recorded_at=NOW - timedelta(minutes=m)) for m in range(60)]) recorded_at=NOW - timedelta(minutes=m)) for m in range(60)])
@@ -137,6 +140,58 @@ def test_a_burst_past_its_usual_peak_is_an_anomaly(app, src):
assert a.started_at == NOW - timedelta(minutes=15) assert a.started_at == NOW - timedelta(minutes=15)
def _window_of(app, src: str, res: str, readings: dict[int, float]):
"""A detection pass at NOW over `readings` ({minutes before NOW: value}),
against the normal that holds 10 and peaks at 20."""
from sqlalchemy import select
from steward.core.anomalies import detect
from steward.models.metrics import PluginMetric, PluginMetricLatest
from steward.models.norms import MetricAnomaly
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add_all(_norms(src, res))
s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC,
value=v, recorded_at=NOW - timedelta(minutes=m))
for m, v in readings.items()])
await s.merge(PluginMetricLatest(source_module=src, resource_name=res, metric_name=METRIC,
value=readings[min(readings)], recorded_at=NOW))
async with s.begin():
await detect(s, NOW)
return list((await s.execute(select(MetricAnomaly).where(MetricAnomaly.resource_name == res))).scalars())
return asyncio.run(_go())
def _every_3_minutes(value: float = 10.0) -> dict[int, float]:
"""Six readings across the 15-minute window, as a router or monitor reports."""
return {m: value for m in range(0, 15, 3)} | {14: value}
def test_one_slow_reading_is_no_anomaly(app, src):
# One 200 among five 10s averages 41.7, past the peak of 20; without it the window holds 10.
assert _window_of(app, src, _res("one"), _every_3_minutes() | {6: 200.0}) == []
def test_two_slow_readings_past_the_peak_are_an_anomaly(app, src):
(a,) = _window_of(app, src, _res("two"), _every_3_minutes() | {3: 200.0, 6: 200.0})
assert a.started_at == NOW - timedelta(minutes=15)
def test_a_lone_reading_is_no_anomaly(app, src):
# A quiet router's only request in the hour: neither a height nor a held level.
assert _window_of(app, src, _res("lone"), {2: 500.0}) == []
def test_a_burst_past_its_slots_peak_but_within_what_most_hours_reach_is_no_anomaly(app, src):
assert _hour_of(app, src, _res("anywhere"), level=30.0, for_minutes=15, anywhere=40.0) == []
def test_a_level_within_what_most_hours_reach_is_an_anomaly_once_held(app, src):
(a,) = _hour_of(app, src, _res("anyheld"), level=30.0, for_minutes=45, anywhere=40.0)
assert a.peak_level == 30.0
def test_a_level_within_its_usual_peak_is_an_anomaly_once_held(app, src): def test_a_level_within_its_usual_peak_is_an_anomaly_once_held(app, src):
# The system running at its peaked rate: never past the peak, but not a burst. # The system running at its peaked rate: never past the peak, but not a burst.
(a,) = _hour_of(app, src, _res("held"), level=18.0, for_minutes=45) (a,) = _hour_of(app, src, _res("held"), level=18.0, for_minutes=45)
@@ -374,6 +429,42 @@ def test_a_metric_learns_only_from_its_since(app, src):
assert held_back is None assert held_back is None
def test_bursts_at_any_hour_are_told_from_a_job_with_a_time_of_its_own(app, src):
"""Fourteen days of a metric at 0. One burst to 50 in each hour of the day
leaves every slot's usual peak at 0 and is usual anywhere. The same burst
every night at 02:00 is usual at 02:00, and still stands out at 14:00."""
from steward.core.norms import norms_at, rebuild_norms
from steward.models.metrics import PluginMetricHourly
sporadic, nightly = _res("sporadic"), _res("nightly")
midnight = NOW.replace(hour=0)
def hours(res, burst):
return [PluginMetricHourly(source_module=src, resource_name=res, metric_name=METRIC,
bucket=midnight - timedelta(days=d) + timedelta(hours=h),
value_avg=0.0, value_p95=0.0, sample_count=60,
value_max=50.0 if burst(d, h) else 0.0)
for d in range(1, 15) for h in range(24)]
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add_all(hours(sporadic, lambda d, h: h % 14 == d - 1))
s.add_all(hours(nightly, lambda d, h: h == 2))
async with s.begin():
await rebuild_norms(s, NOW)
at = {}
for hour in (2, 14):
at[hour] = await norms_at(s, [(src, sporadic, METRIC), (src, nightly, METRIC)],
NOW.replace(hour=hour))
return at
at = asyncio.run(_go())
assert at[14][(src, sporadic, METRIC)].usual_peak == 0.0
assert at[14][(src, sporadic, METRIC)].usual_height == 50.0
assert at[2][(src, nightly, METRIC)].usual_height == 50.0
assert at[14][(src, nightly, METRIC)].usual_height == 0.0
def test_a_metric_learned_without_its_parts_ignores_them(app, src): def test_a_metric_learned_without_its_parts_ignores_them(app, src):
"""A host's CPU learns a normal and each of its cores doesn't (#5742): """A host's CPU learns a normal and each of its cores doesn't (#5742):
with parts=False, "<host>:core0" is neither learned nor judged, and the with parts=False, "<host>:core0" is neither learned nor judged, and the
+187
View File
@@ -0,0 +1,187 @@
"""Integration: a swarm service's metrics stay under one key across its tasks,
and history kept under a task's own name moves onto that key in the background.
Requires STEWARD_DATABASE_URL."""
from __future__ import annotations
import asyncio
import importlib
import os
import uuid
from datetime import datetime, timedelta, timezone
import pytest
pytestmark = pytest.mark.integration
NOW = datetime(2026, 10, 12, 10, 0, tzinfo=timezone.utc)
TASK = "a" * 25
@pytest.fixture
def app():
if not os.environ.get("STEWARD_DATABASE_URL"):
pytest.skip("needs Postgres")
from steward.app import create_app
return create_app(testing=False)
def _docker(sub):
"""The app's loaded docker plugin module (see test_docker_recent_usage)."""
from steward.core.plugin_manager import get_loaded_plugin_modules
loaded = get_loaded_plugin_modules().get("docker")
base = loaded.__name__ if loaded is not None else "plugins.docker"
return importlib.import_module(f"{base}.{sub}")
def _hour(res, metric, bucket, *, avg, top, count=60):
from steward.models.metrics import PluginMetricHourly
return PluginMetricHourly(source_module="docker", resource_name=res, metric_name=metric, bucket=bucket,
value_avg=avg, value_max=top, value_p95=top, sample_count=count)
def _anomaly(res, metric, at, *, peak):
from steward.models.norms import MetricAnomaly
return MetricAnomaly(source_module="docker", resource_name=res, metric_name=metric, started_at=at,
last_started_at=at, ended_at=at + timedelta(minutes=30), occurrences=1,
peak_level=peak, peak_at=at, usual_hold=1.0, usual_peak=2.0)
def test_a_replica_or_a_global_task_keys_by_its_service_and_a_plain_container_by_its_host(app):
service = _docker("service")
assert service.container_resource("nas", f"stack_web.1.{TASK}") == "swarm/stack_web"
assert service.container_resource("nas", f"stack_agent.{'b' * 25}.{TASK}") == "swarm/stack_agent"
assert service.container_resource("nas", "plex") == "nas/plex"
assert service.container_resource("nas", "my.app.1") == "nas/my.app.1"
assert service.traffic_resource("nas", f"stack_web.1.{TASK}") == "nas/stack_web"
assert service.traffic_resource("nas", "plex") == "nas/plex"
def test_a_tasks_history_moves_onto_its_service_a_batch_at_a_time(app, monkeypatch):
from sqlalchemy import select
from steward.models.metrics import PluginMetric, PluginMetricHourly, PluginMetricLatest
from steward.models.norms import MetricAnomaly, MetricAnomalySpan
rekey = _docker("rekey")
monkeypatch.setattr(rekey, "BATCH", 2)
name = f"it{uuid.uuid4().hex[:8]}_web"
old, new, traffic = f"nas/{name}.1.{TASK}", f"swarm/{name}", "net_rx_bps"
shared, alone = NOW - timedelta(hours=3), NOW - timedelta(hours=2)
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add_all([PluginMetric(source_module="docker", resource_name=old, metric_name="cpu_pct",
value=float(m), recorded_at=NOW - timedelta(minutes=m)) for m in range(3)])
s.add(PluginMetric(source_module="docker", resource_name=old, metric_name=traffic,
value=5.0, recorded_at=NOW))
s.add_all([_hour(old, "cpu_pct", shared, avg=10.0, top=30.0),
_hour(new, "cpu_pct", shared, avg=20.0, top=25.0),
_hour(old, "cpu_pct", alone, avg=7.0, top=8.0)])
s.add(PluginMetricLatest(source_module="docker", resource_name=old, metric_name="cpu_pct",
value=1.0, recorded_at=NOW))
older = _anomaly(old, "cpu_pct", NOW - timedelta(days=2), peak=90.0)
newer = _anomaly(new, "cpu_pct", NOW - timedelta(days=1), peak=40.0)
s.add_all([older, newer])
await s.flush()
s.add(MetricAnomalySpan(anomaly_id=older.id, started_at=older.started_at,
ended_at=older.ended_at, peak_level=90.0))
async with s.begin():
found = await rekey.old_keys(s)
async with s.begin():
first = await rekey.move_key(s, old)
async with s.begin():
second = await rekey.move_key(s, old)
raw = (await s.execute(select(PluginMetric.resource_name, PluginMetric.metric_name).where(
PluginMetric.resource_name.in_([old, new])))).all()
hours = {(h.resource_name, h.bucket): h for h in (await s.execute(select(PluginMetricHourly).where(
PluginMetricHourly.resource_name.in_([old, new])))).scalars()}
latest = (await s.execute(select(PluginMetricLatest).where(
PluginMetricLatest.resource_name == old))).scalars().all()
rows = (await s.execute(select(MetricAnomaly).where(
MetricAnomaly.resource_name.in_([old, new])))).scalars().all()
spans = (await s.execute(select(MetricAnomalySpan).where(
MetricAnomalySpan.anomaly_id.in_([r.id for r in rows])))).scalars().all()
left = await rekey.old_keys(s)
return found, first, second, raw, hours, latest, rows, spans, left
found, first, second, raw, hours, latest, rows, spans, left = asyncio.run(_go())
assert old in found and old not in left
assert (first, second) == (False, True)
assert sorted(raw) == sorted([(new, "cpu_pct")] * 3 + [(old, traffic)])
assert set(hours) == {(new, shared), (new, alone)}
combined = hours[(new, shared)]
assert combined.value_avg == 15.0 and combined.value_max == 30.0 and combined.sample_count == 120
assert hours[(new, alone)].value_avg == 7.0
assert latest == []
(row,) = rows
assert row.resource_name == new and row.occurrences == 2 and row.peak_level == 90.0
assert row.started_at == NOW - timedelta(days=2)
assert [sp.anomaly_id for sp in spans] == [row.id]
def test_keys_with_raw_samples_are_taken_first_and_a_pass_finishes_many_small_keys(app):
"""Taken from the end of old_keys: a chart and a normal read recent history,
so a key with raw samples doesn't wait behind thousands that only have old
hours. One pass moves every small key it has time for."""
from sqlalchemy import select
from steward.models.metrics import PluginMetricHourly, PluginMetricLatest
rekey = _docker("rekey")
tag = uuid.uuid4().hex[:8]
aged = [f"nas/it{tag}_old.1.{c * 25}" for c in "bcd"]
recent = f"nas/it{tag}_new.1.{TASK}"
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add_all([_hour(k, "cpu_pct", NOW, avg=1.0, top=2.0) for k in aged])
s.add(PluginMetricLatest(source_module="docker", resource_name=recent, metric_name="cpu_pct",
value=1.0, recorded_at=NOW))
async with s.begin():
found = await rekey.old_keys(s)
await rekey.make_rekey_task(app).coro_factory()
async with app.db_sessionmaker() as s:
left = (await s.execute(select(PluginMetricHourly.resource_name).where(
PluginMetricHourly.resource_name.in_(aged)))).scalars().all()
moved = (await s.execute(select(PluginMetricHourly.value_avg).where(
PluginMetricHourly.resource_name == f"swarm/it{tag}_old"))).scalars().all()
return found, left, moved
found, left, moved = asyncio.run(_go())
assert all(found.index(recent) > found.index(k) for k in aged)
assert left == []
assert moved == [1.0]
def test_an_alert_rule_on_a_task_follows_it_to_its_service(app):
from sqlalchemy import select
from steward.models.alerts import AlertOperator, AlertRule
from steward.models.users import User, UserRole
rekey = _docker("rekey")
tag = uuid.uuid4().hex[:8]
old, new = f"nas/it{tag}_db.1.{TASK}", f"swarm/it{tag}_db"
uid = str(uuid.uuid4())
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add(User(id=uid, username=f"u{tag}", email=f"{tag}@example.com",
password_hash="x", role=UserRole.viewer, is_active=True))
await s.flush()
s.add(AlertRule(name=f"busy {tag}", source_module="docker", resource_name=old,
metric_name="cpu_pct", operator=AlertOperator(">"), threshold=90,
created_by=uid))
async with s.begin():
found = await rekey.old_keys(s)
async with s.begin():
done = await rekey.move_key(s, old)
names = (await s.execute(select(AlertRule.resource_name).where(
AlertRule.name == f"busy {tag}"))).scalars().all()
return found, done, names
found, done, names = asyncio.run(_go())
assert old in found and done
assert names == [new]
@@ -59,7 +59,7 @@ def test_a_service_page_reads_every_task(app):
def sample(cname, mins, cpu, mem): def sample(cname, mins, cpu, mem):
when = now - timedelta(minutes=mins) when = now - timedelta(minutes=mins)
return [PluginMetric(source_module="docker", resource_name=f"dock-{tag}/{cname}", return [PluginMetric(source_module="docker", resource_name=f"swarm/{cname.split('.')[0]}",
metric_name=metric, value=value, recorded_at=when) metric_name=metric, value=value, recorded_at=when)
for metric, value in (("cpu_pct", cpu), ("mem_used_bytes", mem))] for metric, value in (("cpu_pct", cpu), ("mem_used_bytes", mem))]
+827
View File
@@ -0,0 +1,827 @@
"""Integration: Steward signs in to Home Assistant, reads it, and installs one
update only after a newer backup, following it to its end. A fake Home Assistant
answers over httpx's mock transport. Requires STEWARD_DATABASE_URL."""
from __future__ import annotations
import asyncio
import importlib
import json
import os
import time
import uuid
from datetime import datetime, timezone
from urllib.parse import parse_qs, urlparse
import httpx
import pytest
from sqlalchemy import delete, select
pytestmark = pytest.mark.integration
ADDON = "update.example_update"
BEFORE, AFTER = "2026-10-30T03:00:00+00:00", "2026-10-31T03:00:00+00:00"
@pytest.fixture
def app():
if not os.environ.get("STEWARD_DATABASE_URL"):
pytest.skip("needs Postgres")
from steward.app import create_app
return create_app(testing=False)
def _ha(sub):
from steward.core.plugin_manager import get_loaded_plugin_modules
loaded = get_loaded_plugin_modules().get("homeassistant")
base = loaded.__name__ if loaded is not None else "plugins.homeassistant"
return importlib.import_module(f"{base}.{sub}")
class FakeHomeAssistant:
"""Its entities, the actions it received, what it refuses, and its sign-ins."""
def __init__(self):
self.calls = []
self.tokens = set()
self.refreshes = []
self.revoked = []
self.refuse = False
self.down = False
self.admin = True
self.reject_install = None
self.reject_reboot = None
self.cut_off = False
self.entities = {
ADDON: self._update("on", "1.0"),
"sensor.backup_last_successful_automatic_backup": {"state": BEFORE, "attributes": {}},
"event.backup_automatic_backup": {"state": BEFORE, "attributes": {"event_type": "completed"}},
}
def _update(self, state, installed):
return {"state": state, "attributes": {"title": "Example", "installed_version": installed,
"latest_version": "1.1", "in_progress": False,
"entity_picture": "/api/hassio/addons/example/icon"}}
def backup_lands(self):
self.entities["sensor.backup_last_successful_automatic_backup"]["state"] = AFTER
def update_lands(self):
self.entities[ADDON] = self._update("off", "1.1")
def handler(self, request: httpx.Request) -> httpx.Response:
if self.down:
raise httpx.ConnectError("connection refused")
path = request.url.path
if path == "/auth/token":
form = parse_qs(request.content.decode())
self.refreshes.append(form)
if self.refuse:
return httpx.Response(400, json={"error": "invalid_grant"})
return httpx.Response(200, json={"access_token": f"access-{len(self.refreshes)}", "expires_in": 1800,
"refresh_token": "refresh-new", "token_type": "Bearer"})
if path == "/auth/revoke":
self.revoked.append(parse_qs(request.content.decode())["token"][0])
return httpx.Response(200)
self.tokens.add(request.headers.get("authorization"))
if self.refuse:
return httpx.Response(401)
if path == "/api/template":
return httpx.Response(200, text="1") if self.admin else httpx.Response(401)
if path.startswith("/api/services/"):
self.calls.append((path.removeprefix("/api/services/"), json.loads(request.content)))
if self.cut_off:
raise httpx.ReadTimeout("no answer yet")
if self.reject_install and path.endswith("update/install"):
return httpx.Response(500, json={"message": self.reject_install})
if self.reject_reboot and path.endswith("hassio/host_reboot"):
return httpx.Response(500, json={"message": self.reject_reboot})
return httpx.Response(200, json=[])
if path == "/api/states":
return httpx.Response(200, json=[{"entity_id": k, **v} for k, v in self.entities.items()])
entity = self.entities.get(path.removeprefix("/api/states/"))
return httpx.Response(200, json=entity) if entity else httpx.Response(404)
@property
def transport(self):
return httpx.MockTransport(self.handler)
BACKUP = ("backup/create_automatic", {})
INSTALL = ("update/install", {"entity_id": ADDON})
async def _clean(app, **connection):
from steward.core.crypto import encrypt_secret
m = _ha("models")
async with app.db_sessionmaker() as s:
async with s.begin():
await s.execute(delete(m.HomeAssistantUpdate))
await s.execute(delete(m.HomeAssistantConnection))
await s.execute(delete(m.HomeAssistantSchedule))
s.add(m.HomeAssistantConnection(**{
"url": "http://ha.test:8123", "client_id": "http://steward.test/",
"refresh_token": encrypt_secret(f"refresh-{uuid.uuid4().hex}"), "can_install": True, **connection}))
async def _state(app):
m = _ha("models")
async with app.db_sessionmaker() as s:
connection = (await s.execute(select(m.HomeAssistantConnection))).scalars().first()
runs = (await s.execute(select(m.HomeAssistantUpdate))).scalars().all()
return connection, runs
def test_a_poll_stores_what_it_read_and_records_its_metrics(app):
from steward.models.metrics import PluginMetricLatest
sched = _ha("scheduler")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
connection, _ = await _state(app)
async with app.db_sessionmaker() as s:
rows = (await s.execute(select(PluginMetricLatest).where(
PluginMetricLatest.source_module == "homeassistant"))).scalars().all()
return connection, {r.metric_name: r.value for r in rows}
connection, metrics = asyncio.run(go())
assert ha.tokens == {"Bearer access-1"}
assert connection.last_error is None and not connection.refused
assert [u["entity_id"] for u in connection.updates] == [ADDON]
assert connection.backup["last_ok"] == BEFORE
assert metrics["is_up"] == 1.0 and metrics["updates_waiting"] == 1.0 and metrics["backup_age_hours"] >= 0
def test_a_sign_in_lasts_until_its_access_token_nearly_runs_out_then_is_refreshed(app):
client = _ha("client")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
connection = (await _state(app))[0]
first = await client.access_token(connection, ha.transport)
again = await client.access_token(connection, ha.transport)
assert first == again and len(ha.refreshes) == 1
for key, (token, _until) in list(client._access.items()):
client._access[key] = (token, time.monotonic() - 1)
return await client.access_token(connection, ha.transport)
assert asyncio.run(go()) == "access-2"
[form] = ha.refreshes[:1]
assert form["grant_type"] == ["refresh_token"] and form["client_id"] == ["http://steward.test/"]
def test_a_refused_sign_in_is_asked_once_and_not_again(app):
sched = _ha("scheduler")
ha = FakeHomeAssistant()
ha.refuse = True
seen = []
real = ha.handler
def counted(request):
seen.append(request.url.path)
return real(request)
transport = httpx.MockTransport(counted)
async def go():
await _clean(app)
await sched.poll(app, transport=transport)
await sched.poll(app, transport=transport)
return (await _state(app))[0]
connection = asyncio.run(go())
assert seen == ["/auth/token"]
assert connection.refused and "Sign in again" in connection.last_error
def test_an_update_waits_for_a_newer_backup_then_installs_and_ends_on_the_timeline(app):
from steward.models.timeline import TimelineEntry
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
assert ha.calls == [BACKUP]
assert run.state == "backing_up" and run.from_version == "1.0" and run.to_version == "1.1"
# No newer backup yet: nothing is installed, however often it is asked.
await upd.advance(app, transport=ha.transport)
await upd.advance(app, transport=ha.transport)
assert ha.calls == [BACKUP]
ha.backup_lands()
await upd.advance(app, transport=ha.transport)
_, [row] = await _state(app)
assert row.state == "installing" and row.backup_at is not None
await upd.advance(app, transport=ha.transport)
assert ha.calls == [BACKUP, INSTALL]
# Home Assistant restarting is expected, and isn't the end of the update.
ha.down = True
await upd.advance(app, transport=ha.transport)
_, [row] = await _state(app)
assert row.state == "installing"
ha.down = False
ha.update_lands()
await upd.advance(app, transport=ha.transport)
connection, [row] = await _state(app)
assert row.state == "done" and row.finished_at is not None
assert connection.last_polled_at is None # the page shows the new version at the next tick
await upd.advance(app, transport=ha.transport)
assert len(ha.calls) == 2
async with app.db_sessionmaker() as s:
entries = (await s.execute(select(TimelineEntry).where(
TimelineEntry.ref_type == "homeassistant_update", TimelineEntry.ref_id == row.id))).scalars().all()
return entries
[entry] = asyncio.run(go())
assert entry.type == "homeassistant.updated" and entry.actor == "someone"
assert "1.0 to 1.1" in entry.summary and entry.area == "service"
def test_a_failed_backup_installs_nothing(app):
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
ha.entities["event.backup_automatic_backup"] = {
"state": AFTER, "attributes": {"event_type": "failed", "failed_reason": "upload_failed"}}
await upd.advance(app, transport=ha.transport)
return (await _state(app))[1]
[run] = asyncio.run(go())
assert run.state == "failed" and "upload_failed" in run.message and "Nothing was installed" in run.message
assert ha.calls == [BACKUP]
def test_an_action_home_assistant_is_still_working_on_counts_as_sent(app):
"""It answers when finished, and an update that restarts it never does."""
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
ha.cut_off = True
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
assert run.state == "backing_up"
ha.backup_lands()
await upd.advance(app, transport=ha.transport)
await upd.advance(app, transport=ha.transport)
await upd.advance(app, transport=ha.transport)
return (await _state(app))[1]
[run] = asyncio.run(go())
assert run.state == "installing" and run.install_sent_at is not None
assert ha.calls == [BACKUP, INSTALL]
def test_an_install_home_assistant_rejects_fails_with_what_it_said(app):
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
ha.reject_install = "Not enough free space"
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
ha.backup_lands()
await upd.advance(app, transport=ha.transport)
await upd.advance(app, transport=ha.transport)
return (await _state(app))[1]
[run] = asyncio.run(go())
assert run.state == "failed" and "Not enough free space" in run.message
def test_an_install_that_couldnt_reach_home_assistant_is_asked_for_again(app):
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
# Now, as the install's deadline counts from the backup while it is unsent.
ha.entities["sensor.backup_last_successful_automatic_backup"]["state"] = datetime.now(timezone.utc).isoformat()
await upd.advance(app, transport=ha.transport)
ha.down = True
await upd.advance(app, transport=ha.transport)
assert (await _state(app))[1][0].install_sent_at is None
ha.down = False
await upd.advance(app, transport=ha.transport)
return (await _state(app))[1]
[run] = asyncio.run(go())
assert run.install_sent_at is not None and ha.calls == [BACKUP, INSTALL]
def test_an_install_past_its_deadline_fails_and_only_one_runs_at_a_time(app):
from datetime import timedelta
sched, upd, m = _ha("scheduler"), _ha("updates"), _ha("models")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport)
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
with pytest.raises(upd.CannotStart, match="under way"):
await upd.start_update(app, ADDON, "someone else", transport=ha.transport)
ha.backup_lands()
await upd.advance(app, transport=ha.transport)
await upd.advance(app, transport=ha.transport)
async with app.db_sessionmaker() as s:
async with s.begin():
row = await s.get(m.HomeAssistantUpdate, run.id)
row.install_sent_at -= timedelta(seconds=upd.INSTALL_DEADLINE_S["addon"] + 1)
await upd.advance(app, transport=ha.transport)
return (await _state(app))[1]
[run] = asyncio.run(go())
assert run.state == "failed" and "Still not on 1.1" in run.message
assert len(ha.calls) == 2
@pytest.mark.parametrize("entity,change,says", [
("update.unknown", None, "isn't a Core, OS or add-on"),
(ADDON, "no_longer_waiting", "no longer waiting"),
(ADDON, "not_an_admin", "isn't an administrator"),
])
def test_nothing_starts_unless_the_update_can_run(app, entity, change, says):
sched, upd = _ha("scheduler"), _ha("updates")
ha = FakeHomeAssistant()
async def go():
await _clean(app, can_install=change != "not_an_admin")
await sched.poll(app, transport=ha.transport)
if change == "no_longer_waiting":
ha.update_lands()
with pytest.raises(upd.CannotStart, match=says):
await upd.start_update(app, entity, "someone", transport=ha.transport)
return (await _state(app))[1]
assert asyncio.run(go()) == []
assert ha.calls == []
def _admin_client(app):
from steward.models.users import User, UserRole
tag = uuid.uuid4().hex[:8]
uid = str(uuid.uuid4())
async def make():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add(User(id=uid, username=f"ha-{tag}", email=f"ha-{tag}@test", password_hash="x",
role=UserRole.admin, is_active=True))
client = app.test_client()
async with client.session_transaction() as sess:
sess.update(user_id=uid, user_role="admin", username=f"ha-{tag}", _csrf="test-token", login_at=time.time())
return client, f"ha-{tag}"
return make()
CSRF = {"X-CSRF-Token": "test-token"}
@pytest.mark.parametrize("admin", [True, False])
def test_signing_in_goes_to_home_assistant_and_keeps_what_comes_back(app, monkeypatch, admin):
from steward.core.crypto import decrypt_secret
client, m = _ha("client"), _ha("models")
ha = FakeHomeAssistant()
ha.admin = admin
for name in ("redeem_code", "is_admin", "revoke"):
real = getattr(client, name)
if name == "revoke":
monkeypatch.setattr(client, name, lambda c, real=real: real(c, ha.transport))
else:
monkeypatch.setattr(client, name, lambda *a, real=real: real(*a, ha.transport))
async def go():
async with app.db_sessionmaker() as s:
async with s.begin():
await s.execute(delete(m.HomeAssistantUpdate))
await s.execute(delete(m.HomeAssistantConnection))
web, username = await _admin_client(app)
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
form={"url": "http://ha.test:8123/", "verify_tls": "on"})
assert response.status_code == 302
to = urlparse(response.headers["Location"])
query = parse_qs(to.query)
assert f"{to.scheme}://{to.netloc}{to.path}" == "http://ha.test:8123/auth/authorize"
assert query["response_type"] == ["code"]
assert urlparse(query["redirect_uri"][0]).netloc == urlparse(query["client_id"][0]).netloc
assert query["redirect_uri"][0].endswith("/plugins/homeassistant/connection/callback")
# Coming back with someone else's state changes nothing.
forged = await web.get("/plugins/homeassistant/connection/callback?code=c&state=forged")
assert forged.status_code == 400 and not ha.refreshes
assert (await _state(app))[0] is None
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
form={"url": "http://ha.test:8123/", "verify_tls": "on"})
state = parse_qs(urlparse(response.headers["Location"]).query)["state"][0]
back = await web.get(f"/plugins/homeassistant/connection/callback?code=the-code&state={state}")
assert back.status_code == 302
again = await web.get(f"/plugins/homeassistant/connection/callback?code=the-code&state={state}")
assert again.status_code == 400 and len(ha.refreshes) == 1 # a state works once
connection = (await _state(app))[0]
removed = await web.post("/plugins/homeassistant/connection/delete", headers=CSRF)
assert removed.status_code == 302
return connection, username, (await _state(app))[0]
connection, username, after = asyncio.run(go())
assert ha.refreshes[0]["grant_type"] == ["authorization_code"] and ha.refreshes[0]["code"] == ["the-code"]
assert connection.url == "http://ha.test:8123" and connection.verify_tls
assert decrypt_secret(connection.refresh_token, context="test") == "refresh-new"
assert connection.refresh_token != "refresh-new"
assert connection.can_install is admin and connection.signed_in_by == username and connection.signed_in_at
assert ha.revoked == ["refresh-new"] and after is None
def test_signing_in_again_ends_the_sign_in_it_replaces(app, monkeypatch):
from steward.core.crypto import encrypt_secret
client = _ha("client")
ha = FakeHomeAssistant()
for name in ("redeem_code", "is_admin"):
real = getattr(client, name)
monkeypatch.setattr(client, name, lambda *a, real=real: real(*a, ha.transport))
real_revoke = client.revoke
monkeypatch.setattr(client, "revoke", lambda c: real_revoke(c, ha.transport))
async def go():
await _clean(app, refresh_token=encrypt_secret("refresh-old"), refused=True)
web, _ = await _admin_client(app)
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
form={"url": "http://ha.test:8123", "verify_tls": "on"})
state = parse_qs(urlparse(response.headers["Location"]).query)["state"][0]
await web.get(f"/plugins/homeassistant/connection/callback?code=c&state={state}")
return (await _state(app))[0]
connection = asyncio.run(go())
assert ha.revoked == ["refresh-old"] and not connection.refused and connection.last_polled_at is None
# ── The system, over the websocket ───────────────────────────────────────────
class FakeSocket:
"""Home Assistant's websocket: `answers` by question type or Supervisor endpoint,
where one left out is refused as it is for a non-administrator."""
def __init__(self, answers, auth="auth_ok"):
self.answers, self.auth = answers, auth
self.waiting = [{"type": "auth_required"}]
self.asked = []
def __call__(self, url, **options):
self.url = url
return self
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return False
async def send(self, text):
question = json.loads(text)
if question["type"] == "auth":
self.waiting.append({"type": self.auth})
return
key = question.get("endpoint") or question["type"]
self.asked.append(key)
# Something unasked arriving first must be passed over.
self.waiting.append({"type": "event", "id": 9999})
self.waiting.append({"id": question["id"], "success": key in self.answers, "result": self.answers.get(key)})
async def recv(self):
return json.dumps(self.waiting.pop(0))
def test_a_poll_keeps_the_system_problems_and_records_them(app):
from steward.models.metrics import PluginMetricLatest
sched = _ha("scheduler")
ha = FakeHomeAssistant()
socket = FakeSocket({
"get_config": {"recovery_mode": False},
"repairs/list_issues": {"issues": [{"domain": "hassio", "translation_key": "issue_x", "severity": "warning"}]},
"frontend/get_translations": {"resources": {"component.hassio.issues.issue_x.title": "Something's wrong"}},
"/resolution/info": {"unhealthy": [], "unsupported": []},
"/host/info": {"disk_free": 50.0, "disk_total": 100.0},
"/addons": {"addons": [{"slug": "a", "state": "started"}, {"slug": "b", "name": "B", "state": "stopped"}]},
"/addons/b/info": {"boot": "auto"},
})
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport, connect=socket)
connection, _ = await _state(app)
async with app.db_sessionmaker() as s:
rows = (await s.execute(select(PluginMetricLatest).where(
PluginMetricLatest.source_module == "homeassistant"))).scalars().all()
return connection, {r.metric_name: r.value for r in rows}
connection, metrics = asyncio.run(go())
assert socket.url == "ws://ha.test:8123/api/websocket"
assert "/addons/a/info" not in socket.asked # a running add-on isn't asked after
assert [p["title"] for p in connection.health["problems"]] == ["Something's wrong", "B should start on its own and isn't running."]
assert metrics["problems_open"] == 2.0 and metrics["disk_free_pct"] == 50.0 and metrics["update_failed"] == 0.0
def test_the_page_shows_storage_used_of_the_total_with_the_percentage(app):
async def go():
await _clean(app, health={"repairs_read": True, "problems": [], "disk_free_gb": 99.8, "disk_used_gb": 12.0,
"disk_total_gb": 116.5, "disk_free_pct": 85.7, "disk_used_pct": 10.3})
web, _ = await _admin_client(app)
return await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
page = asyncio.run(go())
assert "12.0 of 116.5 GB (10%)" in page
assert "Disk free" not in page
def test_the_page_says_install_leaves_the_reboot_to_you_and_the_schedule_makes_it(app):
sched = _ha("scheduler")
ha = _ha_with_os_update_waiting()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
web, _ = await _admin_client(app)
return await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
page = asyncio.run(go())
assert "It takes effect once you reboot the machine" in page
assert "an OS update reboots the machine, with nobody there to watch" in page
def test_a_report_it_couldnt_read_leaves_the_last_one_and_home_assistant_still_answers(app):
sched = _ha("scheduler")
ha = FakeHomeAssistant()
def unreachable(url, **options):
raise OSError("connection refused")
async def go():
await _clean(app, health={"repairs_read": True, "problems": [{"title": "Old"}]})
await sched.poll(app, transport=ha.transport, connect=unreachable)
return (await _state(app))[0]
connection = asyncio.run(go())
assert connection.last_error is None and not connection.refused
assert connection.health["problems"] == [{"title": "Old"}] and "connection refused" in connection.health["error"]
def test_a_refused_websocket_stops_the_polling_like_any_refusal(app):
sched = _ha("scheduler")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}, auth="auth_invalid"))
return (await _state(app))[0]
connection = asyncio.run(go())
assert connection.refused and "Sign in again" in connection.last_error
# ── Updates on a schedule ────────────────────────────────────────────────────
CORE = "update.home_assistant_core_update"
async def _a_slot_just_passed(app, minutes=10):
from datetime import timedelta
m = _ha("models")
at = datetime.now(timezone.utc) - timedelta(minutes=minutes)
async with app.db_sessionmaker() as s:
async with s.begin():
s.add(m.HomeAssistantSchedule(enabled=True, weekday=at.weekday(), at_time=f"{at:%H:%M}", time_zone="UTC",
window_minutes=240, created_at=at - timedelta(days=30)))
async def _schedule_row(app):
m = _ha("models")
async with app.db_sessionmaker() as s:
return (await s.execute(select(m.HomeAssistantSchedule))).scalars().first()
def test_a_slot_installs_the_add_on_first_then_core_and_stops_at_a_failure(app):
sched, plan = _ha("scheduler"), _ha("schedule")
m = _ha("models")
ha = FakeHomeAssistant()
ha.entities[CORE] = ha._update("on", "1.0")
async def go():
await _clean(app)
await _a_slot_just_passed(app)
await plan.step(app, ha.transport) # the slot fires and asks for a fresh read
assert not (await _state(app))[1] and (await _state(app))[0].last_polled_at is None
await plan.step(app, ha.transport) # nothing starts before that read
assert not (await _state(app))[1]
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
[first] = (await _state(app))[1]
assert first.entity_id == ADDON and first.started_by == "schedule"
await plan.step(app, ha.transport) # one at a time
assert len((await _state(app))[1]) == 1
async with app.db_sessionmaker() as s:
async with s.begin():
run = await s.get(m.HomeAssistantUpdate, first.id)
run.state, run.message = "failed", "It broke."
await plan.step(app, ha.transport)
assert "It broke." in (await _schedule_row(app)).stopped
await plan.step(app, ha.transport)
return (await _state(app))[1]
assert [r.entity_id for r in asyncio.run(go())] == [ADDON]
assert ha.calls == [BACKUP]
def test_a_slot_moves_on_after_a_good_update_and_never_retries_one(app):
sched, plan, updates = _ha("scheduler"), _ha("schedule"), _ha("updates")
ha = FakeHomeAssistant()
ha.entities[CORE] = ha._update("on", "1.0")
async def go():
await _clean(app)
await _a_slot_just_passed(app)
await plan.step(app, ha.transport)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
ha.backup_lands()
await updates.advance(app, ha.transport) # the backup landed
await updates.advance(app, ha.transport) # the install is sent
ha.update_lands()
await updates.advance(app, ha.transport) # it's done, and asks for a fresh read
await plan.step(app, ha.transport)
assert len((await _state(app))[1]) == 1
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
runs = (await _state(app))[1]
return {r.entity_id: r.state for r in runs}
assert asyncio.run(go()) == {ADDON: "done", CORE: "backing_up"}
def test_a_slot_that_cant_start_an_update_says_why_and_asks_no_more(app):
sched, plan = _ha("scheduler"), _ha("schedule")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await _a_slot_just_passed(app)
await plan.step(app, ha.transport)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
del ha.entities["sensor.backup_last_successful_automatic_backup"]
await plan.step(app, ha.transport)
return await _schedule_row(app)
row = asyncio.run(go())
assert "wasn't started" in row.stopped and "no automatic backups" in row.stopped and not ha.calls
def test_a_slot_past_its_window_starts_nothing_and_a_new_schedule_waits_for_its_next(app):
sched, plan = _ha("scheduler"), _ha("schedule")
ha = FakeHomeAssistant()
async def go():
await _clean(app)
await _a_slot_just_passed(app, minutes=300)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
assert not (await _state(app))[1]
await _clean(app)
at = datetime.now(timezone.utc)
await plan.save(app, enabled=True, weekday=at.weekday(), at_time="00:00", time_zone="UTC")
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
return (await _state(app))[1]
assert asyncio.run(go()) == []
assert ha.calls == []
REBOOT = ("hassio/host_reboot", {})
def _ha_with_os_update_waiting():
ha = FakeHomeAssistant()
del ha.entities[ADDON]
ha.entities[_ha("service").OS] = ha._update("on", "1.0")
return ha
async def _scheduled_os_update_written(app, ha):
"""A slot's OS update, followed to where it has been written and needs its reboot."""
sched, plan, updates = _ha("scheduler"), _ha("schedule"), _ha("updates")
await _clean(app)
await _a_slot_just_passed(app)
await plan.step(app, ha.transport)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
ha.backup_lands()
await updates.advance(app, ha.transport) # the backup landed
await updates.advance(app, ha.transport) # the install is sent
# Home Assistant reports the new version as installed before the reboot.
ha.entities[_ha("service").OS] = ha._update("off", "1.1")
await updates.advance(app, ha.transport)
[run] = (await _state(app))[1]
assert run.state == "rebooting" and REBOOT not in ha.calls
def test_a_scheduled_os_update_reboots_once_and_is_done_when_the_new_version_runs(app):
updates = _ha("updates")
ha = _ha_with_os_update_waiting()
def down(url, **options):
raise OSError("connection refused")
async def go():
await _scheduled_os_update_written(app, ha)
await updates.advance(app, ha.transport)
assert ha.calls.count(REBOOT) == 1
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.0", "version_pending": "1.1"}}))
assert (await _state(app))[1][0].state == "rebooting" # not gone down yet
web, _ = await _admin_client(app)
page = await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
assert "rebooting into 1.1" in page and 'hx-trigger="every 15s"' in page
await updates.advance(app, ha.transport, down)
assert (await _state(app))[1][0].state == "rebooting" # down, and expected to be
await updates.advance(app, ha.transport, FakeSocket({}))
assert (await _state(app))[1][0].state == "rebooting" # back, but the Supervisor isn't answering yet
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.1", "version_pending": None}}))
return (await _state(app))[1][0]
run = asyncio.run(go())
assert run.state == "done" and run.finished_at is not None
assert ha.calls.count(REBOOT) == 1
def test_a_machine_that_falls_back_to_the_old_os_fails_at_the_deadline_and_ends_the_slot(app):
from datetime import timedelta
updates, plan, m = _ha("updates"), _ha("schedule"), _ha("models")
ha = _ha_with_os_update_waiting()
old_os = FakeSocket({"/os/info": {"version": "1.0", "version_pending": None}})
async def go():
await _scheduled_os_update_written(app, ha)
await updates.advance(app, ha.transport)
await updates.advance(app, ha.transport, old_os)
assert (await _state(app))[1][0].state == "rebooting"
async with app.db_sessionmaker() as s:
async with s.begin():
run = (await s.execute(select(m.HomeAssistantUpdate))).scalars().first()
run.reboot_sent_at = datetime.now(timezone.utc) - timedelta(minutes=updates.REBOOT_DEADLINE_S // 60 + 1)
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.0", "version_pending": None}}))
await _ha("scheduler").poll(app, transport=ha.transport, connect=FakeSocket({}))
await plan.step(app, ha.transport)
return (await _state(app))[1][0], await _schedule_row(app)
run, schedule = asyncio.run(go())
assert run.state == "failed" and "isn't running" in run.message
assert "isn't running" in schedule.stopped
def test_a_refused_reboot_fails_the_update_and_says_it_still_needs_one(app):
updates = _ha("updates")
ha = _ha_with_os_update_waiting()
ha.reject_reboot = "Not now"
async def go():
await _scheduled_os_update_written(app, ha)
await updates.advance(app, ha.transport)
return (await _state(app))[1][0]
run = asyncio.run(go())
assert run.state == "failed" and "Not now" in run.message and "needs a reboot" in run.message
def test_an_os_update_started_by_a_person_leaves_the_reboot_to_them(app):
sched, updates = _ha("scheduler"), _ha("updates")
ha = _ha_with_os_update_waiting()
async def go():
await _clean(app)
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
await updates.start_update(app, _ha("service").OS, "someone", ha.transport)
ha.backup_lands()
await updates.advance(app, ha.transport)
await updates.advance(app, ha.transport)
ha.entities[_ha("service").OS] = ha._update("off", "1.1")
await updates.advance(app, ha.transport)
await updates.advance(app, ha.transport)
return (await _state(app))[1][0]
assert asyncio.run(go()).state == "done"
assert REBOOT not in ha.calls
+1 -1
View File
@@ -82,7 +82,7 @@ def test_an_invite_is_emailed_and_the_invitee_makes_their_own_account_once(app,
assert resp.status_code == 302 and resp.headers["Location"].endswith("?done=invited") assert resp.status_code == 302 and resp.headers["Location"].endswith("?done=invited")
assert len(sent) == 1 and sent[0]["To"] == to assert len(sent) == 1 and sent[0]["To"] == to
text = sent[0].get_body(("plain",)).get_content() text = sent[0].get_body(("plain",)).get_content()
assert f"{aname} invited you to Steward" in text and "EDT" in text and "UTC" in text assert f"{aname} invited you to Steward" in text and re.search(r"\bE[DS]T\b", text) and "UTC" in text
key = _KEY.search(text).group(1) key = _KEY.search(text).group(1)
page = await (await admin.get(resp.headers["Location"])).get_data(as_text=True) page = await (await admin.get(resp.headers["Location"])).get_data(as_text=True)
@@ -134,3 +134,38 @@ def test_a_scrape_records_series_rates_and_a_down_target_fires_its_rule(app):
assert rows[down].last_ok_at is None assert rows[down].last_ok_at is None
assert down_latest[(down, "is_up")] == 0.0 assert down_latest[(down, "is_up")] == 0.0
assert state.state == AlertStateEnum.firing assert state.state == AlertStateEnum.firing
def test_a_refused_token_is_asked_once(app):
"""Home Assistant bans an address after a few refusals, so a refused token
isn't retried at each interval, however long ago the last scrape was."""
from sqlalchemy import select
scheduler = _prom("scheduler")
PrometheusTarget = _prom("models").PrometheusTarget
TokenRefused = _prom("scraper").TokenRefused
name = f"refused-{uuid.uuid4().hex[:8]}"
asked = []
async def fake_scrape(url, token, verify_tls):
asked.append(url)
raise TokenRefused("Answered 401: the token was refused.")
async def go():
async with app.db_sessionmaker() as s:
async with s.begin():
s.add(PrometheusTarget(name=name, url=f"http://{name}/metrics"))
await scheduler.scrape_due(app, scrape=fake_scrape)
async with app.db_sessionmaker() as s:
async with s.begin():
row = (await s.execute(select(PrometheusTarget).where(PrometheusTarget.name == name))).scalar_one()
refused = row.refused
row.last_scraped_at = row.last_scraped_at.replace(year=2000)
await scheduler.scrape_due(app, scrape=fake_scrape)
async with app.db_sessionmaker() as s:
async with s.begin():
await s.delete((await s.execute(select(PrometheusTarget).where(PrometheusTarget.name == name))).scalar_one())
return refused
assert asyncio.run(go()) is True
assert asked == [f"http://{name}/metrics"]
+83
View File
@@ -0,0 +1,83 @@
"""get_thresholds / set_thresholds: the cutoffs host readings are graded by,
read and changed as Settings → Thresholds does."""
from __future__ import annotations
from contextlib import asynccontextmanager
from types import SimpleNamespace as NS
import pytest
from steward.mcp.tools import thresholds
from steward.models.users import UserRole
@pytest.fixture
def store(monkeypatch):
"""The settings table, the app and the audit log, in memory."""
saved: dict = {"thresholds.temp_crit": 85}
app = NS(config={})
audited: list = []
class _Db:
@asynccontextmanager
async def begin(self):
yield
@asynccontextmanager
async def db_session():
yield _Db()
async def get_all_settings(_db):
return dict(saved)
async def set_setting(_db, key, value):
saved[key] = value
async def audit_mcp(*args):
audited.append(args)
monkeypatch.setattr(thresholds, "db_session", db_session)
monkeypatch.setattr(thresholds, "get_all_settings", get_all_settings)
monkeypatch.setattr(thresholds, "set_setting", set_setting)
monkeypatch.setattr(thresholds, "mcp_app", lambda: app)
monkeypatch.setattr(thresholds, "audit_mcp", audit_mcp)
return NS(saved=saved, app=app, audited=audited)
async def test_the_cutoffs_read_with_their_units_and_defaults(store):
got = await thresholds.get_thresholds()
assert set(got) == {"cpu", "mem", "disk", "load", "temp"}
assert got["temp"] == {"warn": 70, "crit": 85, "unit": "°C", "measures": "hottest real thermometer",
"default": {"warn": 70, "crit": 85}}
async def test_raising_one_cutoff_keeps_the_other_and_grades_by_it_at_once(store):
got = await thresholds.set_thresholds({"temp": {"crit": 90}})
assert got["temp"]["crit"] == 90 and got["temp"]["warn"] == 70
assert store.saved == {"thresholds.temp_crit": 90} # warn isn't written
assert store.app.config["THRESHOLDS"]["temp"]["crit"] == 90
(_action, _kind, _id, detail), = store.audited
assert detail["changes"] == {"thresholds.temp_crit": {"from": 85, "to": 90}}
@pytest.mark.parametrize("cutoffs, refusal", [
({}, "Nothing to change"),
({"fan": {"crit": 1}}, "No reading named fan"),
({"temp": {"high": 90}}, "warn and/or crit"),
({"temp": {}}, "warn and/or crit"),
({"temp": {"crit": "hot"}}, "must be a number"),
({"temp": {"crit": 0}}, "more than 0"),
({"temp": {"crit": 60}}, r"warn \(70\) must be below its crit \(60\)"),
])
async def test_refusals_change_nothing(store, cutoffs, refusal):
with pytest.raises(ValueError, match=refusal):
await thresholds.set_thresholds(cutoffs)
assert store.saved == {"thresholds.temp_crit": 85} and not store.audited
def test_the_tools_are_registered_with_their_roles(app):
mcp = app.mcp_server
for name, role, read_only in (("get_thresholds", UserRole.viewer, True),
("set_thresholds", UserRole.admin, False)):
assert mcp.tool_roles[name] is role, name
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is read_only, name
+16 -1
View File
@@ -312,10 +312,25 @@ def test_get_host_groups_parts_by_kind_and_summarises_cores():
assert got["disk_io"]["items"][0]["device"] == "sda" assert got["disk_io"]["items"][0]["device"] == "sda"
assert [d["device"] for d in got["drive_health"]["items"]] == ["nvme0", "sda"] # failing first assert [d["device"] for d in got["drive_health"]["items"]] == ["nvme0", "sda"] # failing first
assert got["temperatures"]["items"][0] == {"sensor": "Package id 0", "part": "temp:Package id 0", assert got["temperatures"]["items"][0] == {"sensor": "Package id 0", "part": "temp:Package id 0",
"temp_c": 61.0} "temp_c": 61.0, "counted": True}
assert "other" not in got assert "other" not in got
def test_get_host_marks_the_sensors_the_hosts_temperature_leaves_out(monkeypatch):
"""An unwired input reading 115 °C is listed after the real sensors, with
why it doesn't count, so a reader doesn't take it for heat."""
from steward.mcp.tools import hosts
monkeypatch.setattr(hosts, "_UNCOUNTED_SENSORS",
lambda host, temps: {k: "reads a value no wired sensor gives"
for k, c in temps.items() if c >= 100})
got = hosts.agent_parts({"temp:SYSTIN": {"temp_c": 115.0}, "temp:Package id 0": {"temp_c": 61.0},
"temp:nvme": {"temp_c": 40.0}}, "box", limit=10)["temperatures"]["items"]
assert [t["sensor"] for t in got] == ["Package id 0", "nvme", "SYSTIN"]
assert got[0]["counted"] is True and "not_counted_because" not in got[0]
assert got[2]["counted"] is False
assert got[2]["not_counted_because"] == "reads a value no wired sensor gives"
def test_one_silent_agent_is_one_finding(): def test_one_silent_agent_is_one_finding():
"""#5721: VDNT-Docker03's silent agent read as three findings. Its agent- """#5721: VDNT-Docker03's silent agent read as three findings. Its agent-
offline alert speaks for it, the stale-agent finding goes, and its rules offline alert speaks for it, the stale-agent finding goes, and its rules
+252
View File
@@ -0,0 +1,252 @@
"""Reading what Home Assistant reports: which updates Steward may install, how
its backups stand, and what the pages and MCP show."""
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace as NS
import plugins.homeassistant as homeassistant_plugin
from plugins.homeassistant import mcp_tools, schedule, scheduler, service as s
from steward.models.users import UserRole
from tests._templates import render_plugin_template
from tests.test_style_tokens import hex_colours
NOW = datetime(2026, 10, 31, 12, 0, tzinfo=timezone.utc)
def _entity(entity_id, state="on", **attrs):
return {"entity_id": entity_id, "state": state, "attributes": attrs}
def _addon(slug="core_example", state="on"):
return _entity(f"update.{slug}_update", state, title="Example add-on", installed_version="1.0",
latest_version="1.1", entity_picture=f"/api/hassio/addons/{slug}/icon")
def test_only_core_os_and_add_ons_can_be_installed():
assert s.classify(_entity(s.CORE)) == "core"
assert s.classify(_entity(s.OS)) == "os"
assert s.classify(_addon()) == "addon"
assert s.classify(_entity(s.SUPERVISOR, entity_picture="/api/brands/integration/hassio/icon.png")) is None
assert s.classify(_entity("update.switch_firmware", device_class="firmware",
entity_picture="/api/brands/integration/unifi/icon.png")) is None
assert s.classify(_entity("update.custom_locks_update",
entity_picture="https://brands.home-assistant.io/_/locks/icon.png")) is None
def test_a_snapshot_keeps_waiting_updates_first():
snapshot = s.read_snapshot([
_addon("core_quiet", "off"), _entity("update.switch_firmware", title="Switch"),
_addon(), _entity(s.CORE, title="Home Assistant Core"),
_entity("update.gone", "unavailable"),
_entity(s.LAST_BACKUP, "2026-10-31T03:00:00+00:00"), _entity(s.NEXT_BACKUP, "unknown"),
_entity(s.BACKUP_EVENT, "2026-10-31T03:00:00+00:00", event_type="completed"),
_entity("light.kitchen", "on"),
])
assert [u["entity_id"] for u in snapshot["updates"]] == [
s.CORE, "update.core_example_update", "update.switch_firmware", "update.core_quiet_update"]
backup = snapshot["backup"]
assert backup["last_ok"] == "2026-10-31T03:00:00+00:00" and backup["next"] is None
assert backup["last_result"] == "completed"
def test_a_release_link_is_only_kept_when_it_is_a_web_address():
assert s.read_snapshot([_entity(s.CORE, release_url="javascript:alert(1)")])["updates"][0]["release_url"] is None
assert s.read_snapshot([_entity(s.CORE, release_url="https://example.com/r")])["updates"][0]["release_url"]
def test_newer_is_judged_on_the_readings_alone():
assert s.is_newer("2026-10-31T03:00:01+00:00", "2026-10-31T03:00:00+00:00")
assert not s.is_newer("2026-10-31T03:00:00+00:00", "2026-10-31T03:00:00+00:00")
assert not s.is_newer("2026-10-30T03:00:00+00:00", "2026-10-31T03:00:00+00:00")
assert s.is_newer("2026-10-31T03:00:00+00:00", "unknown")
assert not s.is_newer("unavailable", "2026-10-31T03:00:00+00:00")
def test_how_the_backups_read():
ok = {"known": True, "last_ok": (NOW - timedelta(hours=25)).isoformat(), "last_result": "completed"}
assert s.backup_verdict(ok, NOW)[0] == "ok"
assert abs(s.backup_age_hours(ok, NOW) - 25) < 0.01
stale = {**ok, "last_ok": (NOW - timedelta(hours=27)).isoformat()}
assert s.backup_verdict(stale, NOW) == ("warn", "The last good backup is 27 hours old, past the 26 hours expected.")
failed = {**ok, "last_result": "failed", "failed_reason": "upload_failed"}
assert "failed (upload_failed)" in s.backup_verdict(failed, NOW)[1]
assert "no successful" in s.backup_verdict({"known": True, "last_ok": None}, NOW)[1]
assert "reports no automatic backups" in s.backup_verdict({}, NOW)[1]
def test_a_refused_sign_in_stops_polling_until_someone_acts():
fresh = NS(refused=False, last_polled_at=NOW - timedelta(seconds=60))
assert not scheduler.is_due(fresh, NOW)
assert scheduler.is_due(NS(refused=False, last_polled_at=NOW - timedelta(seconds=300)), NOW)
assert scheduler.is_due(NS(refused=False, last_polled_at=None), NOW)
assert not scheduler.is_due(NS(refused=True, last_polled_at=None), NOW)
def _connection(**over):
return NS(**{"url": "http://ha.example:8123", "refresh_token": "tok-s3cret", "can_install": True,
"verify_tls": True, "signed_in_by": "someone", "signed_in_at": NOW,
"refused": False, "last_error": None, "last_polled_at": NOW, "last_ok_at": NOW,
"backup": {"known": True, "last_ok": NOW.isoformat(), "last_result": "completed"},
"updates": [], "health": {}, **over})
def test_the_overview_names_what_needs_a_look_and_no_more():
assert mcp_tools.overview_findings(None, None, NOW) == []
assert mcp_tools.overview_findings(_connection(), None, NOW) == []
waiting = _connection(updates=[{"waiting": True, "kind": "core", "title": "Core"}])
assert mcp_tools.overview_findings(waiting, None, NOW) == []
down = mcp_tools.overview_findings(_connection(last_error="Couldn't reach Home Assistant."), None, NOW)
assert [f["kind"] for f in down] == ["home_assistant_down"]
stale = _connection(backup={"known": True, "last_ok": (NOW - timedelta(days=2)).isoformat()})
assert [f["kind"] for f in mcp_tools.overview_findings(stale, None, NOW)] == ["home_assistant_backup"]
failed = NS(state="failed", title="Core", message="No new backup.", finished_at=NOW)
[f] = mcp_tools.overview_findings(_connection(), failed, NOW)
assert f["kind"] == "home_assistant_update_failed" and "No new backup." in f["summary"]
two = {"problems": [{"title": "Disk trouble"}, {"title": "Another"}]}
[f] = mcp_tools.overview_findings(_connection(health=two), None, NOW)
assert f["kind"] == "home_assistant_problem" and f["summary"] == "Disk trouble (and 1 more)"
plan = NS(stopped="Core: it failed.", last_slot_at=None)
[f] = mcp_tools.overview_findings(_connection(), None, NOW, plan)
assert f["kind"] == "home_assistant_schedule_stopped" and "Core: it failed." in f["summary"]
def test_system_problems_are_read_from_what_home_assistant_answers():
healthy = s.read_health({"config": {"recovery_mode": False}, "issues": [], "resolution": {"unhealthy": [], "unsupported": []},
"host": {"disk_free": 99.8, "disk_used": 12.0, "disk_total": 116.5},
"addons": [{"slug": "a", "state": "started"}]})
assert healthy["problems"] == [] and healthy["repairs_read"] and healthy["supervised"]
assert healthy["disk_free_pct"] == 85.7 and healthy["disk_used_gb"] == 12.0 and healthy["disk_used_pct"] == 10.3
bad = s.read_health({
"config": {"recovery_mode": True},
"issues": [{"domain": "hassio", "translation_key": "issue_system_reboot_required", "severity": "warning"},
{"domain": "demo", "translation_key": "named", "translation_placeholders": {"who": "X"}, "severity": "critical"},
{"domain": "demo", "translation_key": "gone", "ignored": True},
{"domain": "demo", "translation_key": "old", "dismissed_version": "1.0"}],
"titles": {"component.demo.issues.named.title": "{who} broke"},
"resolution": {"unhealthy": ["docker_gateway"], "unsupported": []},
"host": {"disk_free": 5.0, "disk_total": 100.0},
"addons": [{"slug": "should", "name": "Should run", "state": "stopped"}, {"slug": "once", "state": "stopped"}],
"boots": {"should": "auto", "once": "manual"}})
assert [p["kind"] for p in bad["problems"]] == ["recovery", "repair", "repair", "supervisor", "disk", "addon"]
titles = [p["title"] for p in bad["problems"]]
assert titles[1] == "System reboot required" and titles[2] == "X broke" and "docker gateway" in titles[3]
assert "5 GB of 100 GB" in titles[4] and "Should run" in titles[5]
def test_what_home_assistant_wouldnt_answer_is_not_read_as_fine():
found = s.read_health({"config": {}, "issues": None, "resolution": None, "host": None, "addons": None})
assert not found["repairs_read"] and not found["supervised"]
assert found["disk_free_pct"] is None and found["disk_used_pct"] is None
def test_a_schedule_installs_add_ons_then_core_then_the_os():
def update(entity_id, kind, in_progress=False):
return {"entity_id": entity_id, "kind": kind, "in_progress": in_progress}
waiting = [update("os", "os"), update("core", "core"), update("addon", "addon")]
assert schedule.next_to_install(waiting, set())["entity_id"] == "addon"
assert schedule.next_to_install(waiting, {"addon"})["entity_id"] == "core"
assert schedule.next_to_install(waiting, {"addon", "core"})["entity_id"] == "os"
assert schedule.next_to_install(waiting, {"addon", "core", "os"}) is None
assert schedule.next_to_install([update("addon", "addon", in_progress=True)], set()) is None
def test_reading_is_open_to_a_viewer_and_installing_is_for_an_admin():
seen = {}
def register(fn, *, role, read_only): # both required, as on the server
seen[fn.__name__] = (role, read_only)
homeassistant_plugin.register_mcp_tools(register)
assert seen == {"get_home_assistant": (UserRole.viewer, True), "update_home_assistant": (UserRole.admin, False),
"set_home_assistant_schedule": (UserRole.admin, False)}
def _page_values(**over):
update = {"entity_id": s.CORE, "title": "Core", "kind": "core", "kind_word": "Core", "installed": "1.0",
"latest": "1.1", "waiting": True, "in_progress": False, "auto_update": False,
"release_url": "https://example.com/r"}
return {"connection": _connection(), "reach": "up", "says": "Answering.", "installable": [update], "others": [],
"current": [], "runs": [], "under_way": None, "backup": {}, "backup_tone": "ok",
"backup_says": "Backups are up to date.", "backup_at": NOW, "backup_next": None, "backup_age": 1.0,
"stale_hours": 26, "health": {"repairs_read": True, "supervised": True}, "problems": [], "plan": None,
"plan_when": None, "plan_next": None, "weekdays": ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
"Saturday", "Sunday"), "default_zone": "UTC", **over}
async def _render(app, role, **values):
from quart import session
async with app.test_request_context("/plugins/homeassistant/"):
session["user_role"] = role
page = await render_plugin_template(app, "homeassistant", "homeassistant/index.html", **values)
return " ".join(page.split())
async def test_an_admin_is_asked_before_an_install_and_a_viewer_cant_install(app):
page = await _render(app, "admin", **_page_values())
assert "1 update waiting" in page and 'action="/plugins/homeassistant/install"' in page
assert "Steward takes a backup first" in page and "Home Assistant restarts." in page
assert "tok-s3cret" not in page
main = page[page.index("<main"):]
assert not hex_colours(main), hex_colours(main)
assert 'action="/plugins/homeassistant/install"' not in await _render(app, "viewer", **_page_values())
async def test_install_is_held_back_while_it_couldnt_work_and_the_page_says_why(app):
run = NS(id="r", title="Core", state="backing_up", from_version="1.0", to_version="1.1", started_by="someone",
started_at=NOW, message=None)
busy = await _render(app, "admin", **_page_values(under_way=run, runs=[run]))
assert 'action="/plugins/homeassistant/install"' not in busy and "taking a backup first" in busy
assert 'hx-trigger="every 15s"' in busy
status_only = await _render(app, "admin", **_page_values(connection=_connection(can_install=False)))
assert 'action="/plugins/homeassistant/install"' not in status_only
assert "isn't an administrator" in status_only
async def test_the_page_shows_system_problems_and_what_it_couldnt_read(app):
clear = await _render(app, "viewer", **_page_values())
assert "No repairs open" in clear
problem = {"kind": "repair", "severity": "warning", "title": "System reboot required"}
shown = await _render(app, "viewer", **_page_values(problems=[problem]))
assert "System reboot required" in shown and "Settings → System → Repairs" in shown
hidden = await _render(app, "viewer", **_page_values(health={"repairs_read": False, "supervised": False}))
assert "wouldn't share its repairs" in hidden
bare = await _render(app, "viewer", **_page_values(health={"repairs_read": True, "supervised": False}))
assert "No Supervisor answered" in bare
async def test_an_admin_sets_the_schedule_warned_of_what_it_does_and_a_viewer_only_reads_it(app):
off = await _render(app, "admin", **_page_values())
assert 'action="/plugins/homeassistant/schedule"' in off and "Off. Updates are only installed" in off
assert "with nobody there to watch" in off
assert '<option value="5" selected>Saturday</option>' in off and 'value="05:00"' in off
plan = NS(enabled=True, weekday=5, at_time="05:00", time_zone="Europe/London", stopped="Core: it failed.")
values = _page_values(plan=plan, plan_when="Saturdays at 05:00 (Europe/London), up to 4 h", plan_next=NOW)
on = await _render(app, "viewer", **values)
assert "Saturdays at 05:00" in on and "Core: it failed." in on
assert 'action="/plugins/homeassistant/schedule"' not in on
async def test_an_unconnected_page_offers_to_connect(app):
page = await _render(app, "admin", connection=None)
assert 'href="/plugins/homeassistant/connection"' in page
async def _form(app, **values):
from quart import session
async with app.test_request_context("/plugins/homeassistant/connection"):
session["user_role"] = "admin"
page = await render_plugin_template(app, "homeassistant", "homeassistant/form.html", error=None,
url="http://ha.example:8123", verify_tls=True, **values)
return " ".join(page.split())
async def test_the_form_warns_what_the_credential_can_do_and_recommends_a_service_account(app):
page = await _form(app, connection=None)
assert "same access as the user who signs in" in page and "including controlling every device" in page
assert "We recommend a service account" in page and "in Home Assistant's logs" in page
assert 'action="/plugins/homeassistant/connection/sign-in"' in page and "Continue to Home Assistant" in page
assert "connection/delete" not in page
async def test_a_connected_form_says_who_signed_in_and_never_shows_the_credential(app):
page = await _form(app, connection=_connection())
assert "tok-s3cret" not in page and "by someone" in page and "an administrator" in page
assert "Sign in again" in page and "connection/delete" in page
@@ -146,3 +146,19 @@ def test_build_sample_first_call_has_no_rates_but_keeps_state(monkeypatch):
assert s1["net"] == [] and s1["diskio"] == [] # no prior counters yet assert s1["net"] == [] and s1["diskio"] == [] # no prior counters yet
assert state["mono"] is not None assert state["mono"] is not None
assert state["net"] == {"eth0": (1000, 500)} assert state["net"] == {"eth0": (1000, 500)}
def test_collect_temps_says_which_chip_each_sensor_is_on(tmp_path, monkeypatch):
nvme = tmp_path / "hwmon1"
nvme.mkdir()
(nvme / "name").write_text("nvme\n")
(nvme / "temp3_input").write_text("49850\n")
(nvme / "temp3_label").write_text("Sensor 2\n")
bare = tmp_path / "hwmon2"
bare.mkdir()
(bare / "name").write_text("acpitz\n")
(bare / "temp1_input").write_text("27800\n")
monkeypatch.setattr(agent, "HWMON_DIR", str(tmp_path))
got = sorted(agent.collect_temps(), key=lambda t: t["label"])
assert got == [{"label": "Sensor 2", "celsius": 49.9, "chip": "nvme"},
{"label": "acpitz_temp1", "celsius": 27.8, "chip": "acpitz"}]
@@ -114,6 +114,24 @@ def test_the_heat_alert_names_the_sensor_that_set_it():
assert hottest_sensor("box", {"AUXTIN0": 110.0}) == [] assert hottest_sensor("box", {"AUXTIN0": 110.0}) == []
def test_the_hottest_sensor_is_named_by_its_chip_once_the_agent_says_it():
"""An NVMe drive's "Sensor 2" alone doesn't say it's the drive."""
from plugins.host_agent.routes import hottest_sensor
sample = {"ts": "2026-06-16T00:00:00+00:00",
"temps": [{"label": "Sensor 2", "celsius": 49.0, "chip": "nvme"},
{"label": "Package id 0", "celsius": 41.0, "chip": "coretemp"},
{"label": "nct6798_temp9", "celsius": 30.0, "chip": "nct6798"},
{"label": "fan_ic", "celsius": 35.0, "chip": "mystery9"}]}
idx = _index(_expand_sample_to_metrics(sample, "chipbox", datetime.now(timezone.utc)))
assert idx[("temp_c", "chipbox:temp:Sensor 2")] == 49.0 # keyed by label alone
temps = {"Sensor 2": 49.0, "Package id 0": 41.0}
assert hottest_sensor("chipbox", temps) == ["NVMe · Sensor 2 at 49°C"]
assert hottest_sensor("chipbox", {"Package id 0": 41.0}) == ["CPU · Package id 0 at 41°C"]
assert hottest_sensor("chipbox", {"nct6798_temp9": 30.0}) == ["nct6798_temp9 at 30°C"]
assert hottest_sensor("chipbox", {"fan_ic": 35.0}) == ["mystery9 · fan_ic at 35°C"]
assert hottest_sensor("otherbox", temps) == ["Sensor 2 at 49°C"] # an agent that never said
def test_a_wired_board_input_still_counts(): def test_a_wired_board_input_still_counts():
"""#5759: below 100 °C a board input is a real reading.""" """#5759: below 100 °C a board input is a real reading."""
from plugins.host_agent.routes import counts_as_hottest from plugins.host_agent.routes import counts_as_hottest
+28 -6
View File
@@ -25,16 +25,16 @@ def test_worst_reading(readings, worst):
assert worst_reading(readings, TH) == worst assert worst_reading(readings, TH) == worst
async def _vitals(app, *, cpu=40.0, mem=50.0, stale=False, seen_min=1): async def _vitals(app, *, cpu=40.0, mem=50.0, stale=False, seen_min=1, temp=None, hottest=None):
readings = (("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", 30.0, "disk"), readings = (("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", 30.0, "disk"),
("Load per core", 20, "load")) ("Load per core", 20, "load"), ("Temperature", temp, "temp"))
async with app.test_request_context("/plugins/host_agent/vitals/h1"): async with app.test_request_context("/plugins/host_agent/vitals/h1"):
page = await render_plugin_template( page = await render_plugin_template(
app, "host_agent", "_host_vitals.html", reporting=True, stale=stale, app, "host_agent", "_host_vitals.html", reporting=True, stale=stale,
reg=SimpleNamespace(last_seen_at=datetime.now(timezone.utc) - timedelta(minutes=seen_min), reg=SimpleNamespace(host_id="h1", agent_version="1.9.0",
agent_version="1.9.0"), last_seen_at=datetime.now(timezone.utc) - timedelta(minutes=seen_min)),
cpu=cpu, mem=mem, disk_root=30.0, load_per_core=20, cpu=cpu, mem=mem, disk_root=30.0, load_per_core=20, temp=temp, hottest=hottest,
sparks={"cpu": "", "mem": "", "disk": "", "load": ""}, sparks={"cpu": "", "mem": "", "disk": "", "load": "", "temp": "<svg></svg>"},
psi={"cpu": None, "mem": None, "io": None}, worst=worst_reading(readings, TH)) psi={"cpu": None, "mem": None, "io": None}, worst=worst_reading(readings, TH))
return re.sub(r">\s+<", "><", " ".join(page.split())) return re.sub(r">\s+<", "><", " ".join(page.split()))
@@ -53,6 +53,28 @@ async def test_a_quiet_host_says_so(app):
assert "Every reading is within its thresholds." in await _vitals(app) assert "Every reading is within its thresholds." in await _vitals(app)
async def test_the_strip_at_the_top_of_the_page_leads_to_the_full_metrics(app):
"""The temperatures, charts and every mount are on the full-metrics page;
its link sat in the Agent card below the fold and was missed."""
assert 'href="/plugins/host_agent/h1/"' in await _vitals(app)
async def test_the_strip_shows_the_hottest_sensor_with_its_name_and_line(app):
page = await _vitals(app, temp=62.0, hottest="Package id 0")
assert "--tiles:5;" in page
assert ('class="stat-tile stat-ok"><span class="stat-tile-label">Temperature</span>'
'<span class="stat-tile-value">62°C</span>') in page
assert '<svg></svg><span class="stat-tile-sensor" title="Package id 0">Package id 0</span>' in page
without = await _vitals(app)
assert "--tiles:4;" in without and ">Temperature<" not in without
async def test_a_host_past_its_heat_threshold_says_so_in_degrees(app):
page = await _vitals(app, temp=88.0, hottest="Package id 0") # crit at 85
assert '<span class="tone-crit">Temperature 88°C</span> is past its critical threshold.' in page
assert 'class="stat-tile stat-crit"><span class="stat-tile-label">Temperature</span>' in page
async def test_a_stale_host_says_since_when_before_any_reading(app): async def test_a_stale_host_says_since_when_before_any_reading(app):
page = await _vitals(app, mem=93.0, stale=True, seen_min=20) page = await _vitals(app, mem=93.0, stale=True, seen_min=20)
assert '<span class="badge badge-warn">Stale</span>' in page assert '<span class="badge badge-warn">Stale</span>' in page
+15 -1
View File
@@ -270,7 +270,7 @@ def test_a_zypper_update_that_asks_for_a_reboot_still_worked(maint, monkeypatch)
monkeypatch.setattr(maint, "_os_release_id", lambda path="/etc/os-release": "sles") monkeypatch.setattr(maint, "_os_release_id", lambda path="/etc/os-release": "sles")
ran = [] ran = []
def fake_run(cmd, deadline=0, env=None, ok_codes=(0,)): def fake_run(cmd, deadline=0, env=None, ok_codes=(0,), keep=0):
ran.append(cmd[2]) ran.append(cmd[2])
code = 102 if cmd[2] == "update" else 0 code = 102 if cmd[2] == "update" else 0
return code in ok_codes, "3 packages to upgrade.\n" return code in ok_codes, "3 packages to upgrade.\n"
@@ -282,6 +282,20 @@ def test_a_zypper_update_that_asks_for_a_reboot_still_worked(maint, monkeypatch)
assert ran == ["refresh", "update"] assert ran == ["refresh", "update"]
def test_a_large_update_still_counts_its_packages(maint, monkeypatch):
"""apt prints its count before the log of unpacking and setting up each
package, which runs far past what a detail keeps."""
_manager(monkeypatch, maint, ("apt-get",))
log = "17 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n" + \
"Setting up libfoo (1.2-3) ...\n" * 200
monkeypatch.setattr(maint, "run", lambda cmd, **kw: (True, log if kw.get("keep", 0) is None
else log[-maint.OUTPUT_KEPT_CHARS:]))
monkeypatch.setattr(maint, "reboot_required", lambda: False)
got = maint.do_update()
assert got["ok"] and got["packages"] == 17
assert len(got["detail"]) <= 500
def test_a_failed_step_says_which(maint, monkeypatch): def test_a_failed_step_says_which(maint, monkeypatch):
_manager(monkeypatch, maint, ("pacman",)) _manager(monkeypatch, maint, ("pacman",))
monkeypatch.setattr(maint, "run", lambda cmd, **kw: (False, "error: failed to synchronize all databases")) monkeypatch.setattr(maint, "run", lambda cmd, **kw: (False, "error: failed to synchronize all databases"))
+1 -1
View File
@@ -15,7 +15,7 @@ def _target(name, *, enabled=True, scraped=timedelta(seconds=30), error=None,
url="http://archives:8080/metrics"): url="http://archives:8080/metrics"):
return NS(id=f"id-{name}", name=name, url=url, token="bearer-s3cret", enabled=enabled, return NS(id=f"id-{name}", name=name, url=url, token="bearer-s3cret", enabled=enabled,
interval_seconds=60, last_scraped_at=NOW - scraped if scraped else None, interval_seconds=60, last_scraped_at=NOW - scraped if scraped else None,
last_ok_at=NOW - timedelta(hours=1), last_error=error, last_kept=120, last_dropped=30, last_ok_at=NOW - timedelta(hours=1), last_error=error, refused=False, last_kept=120, last_dropped=30,
last_capped=0, series_cap=1000, keep=["archives_*"], drop=[]) last_capped=0, series_cap=1000, keep=["archives_*"], drop=[])
+12 -2
View File
@@ -7,7 +7,7 @@ import pytest
from plugins.prometheus.models import PrometheusTarget from plugins.prometheus.models import PrometheusTarget
from plugins.prometheus.scheduler import is_due from plugins.prometheus.scheduler import is_due
from plugins.prometheus.scraper import ScrapeError, scrape from plugins.prometheus.scraper import ScrapeError, TokenRefused, scrape
from plugins.prometheus.service import (FormError, fold_states, label_pairs, map_families, parse_form, from plugins.prometheus.service import (FormError, fold_states, label_pairs, map_families, parse_form,
patterns, rate_name, resource_for, target_state, wanted) patterns, rate_name, resource_for, target_state, wanted)
from steward.core import prometheus from steward.core import prometheus
@@ -165,6 +165,16 @@ def test_state_and_due():
assert target_state(t, now).reach == "paused" and not is_due(t, now) assert target_state(t, now).reach == "paused" and not is_due(t, now)
def test_a_refused_token_stops_the_scraping_and_the_page_says_so():
now = datetime(2026, 10, 12, 12, 0, tzinfo=timezone.utc)
t = PrometheusTarget(name="a", url="http://a", enabled=True, interval_seconds=60, refused=True,
last_error="Answered 401: the token was refused.",
last_scraped_at=now - timedelta(hours=5))
state = target_state(t, now)
assert state.reach == "down" and "stopped asking" in state.says and "Answered 401" in state.says
assert not is_due(t, now)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_a_scrape_sends_the_token_and_says_why_it_failed(): async def test_a_scrape_sends_the_token_and_says_why_it_failed():
seen = {} seen = {}
@@ -180,7 +190,7 @@ async def test_a_scrape_sends_the_token_and_says_why_it_failed():
transport = httpx.MockTransport(handler) transport = httpx.MockTransport(handler)
fams = await scrape("http://svc/metrics", "tok", transport=transport) fams = await scrape("http://svc/metrics", "tok", transport=transport)
assert list(fams) == ["up_thing"] and seen["auth"] == "Bearer tok" assert list(fams) == ["up_thing"] and seen["auth"] == "Bearer tok"
with pytest.raises(ScrapeError, match="token was refused"): with pytest.raises(TokenRefused, match="token was refused"):
await scrape("http://svc/denied", "bad", transport=transport) await scrape("http://svc/denied", "bad", transport=transport)
with pytest.raises(ScrapeError, match="no metrics"): with pytest.raises(ScrapeError, match="no metrics"):
await scrape("http://svc/html", transport=transport) await scrape("http://svc/html", transport=transport)
+1
View File
@@ -27,6 +27,7 @@ TOP_LEVEL = {
"plugins/traefik/templates/traefik/index.html", "plugins/traefik/templates/traefik/index.html",
"plugins/unifi/templates/unifi/index.html", "plugins/unifi/templates/unifi/index.html",
"plugins/syslog/templates/syslog/index.html", "plugins/syslog/templates/syslog/index.html",
"plugins/homeassistant/templates/homeassistant/index.html",
# Logged out, or no page to go back up from. # Logged out, or no page to go back up from.
"steward/templates/auth/login.html", "steward/templates/auth/login.html",
"steward/templates/auth/setup.html", "steward/templates/auth/setup.html",