Compare commits
21
Commits
177ea12c9b
..
dev
@@ -126,6 +126,7 @@ See `docs/plugins/` for a full plugin development guide.
|
||||
| `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide |
|
||||
| `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples |
|
||||
| `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe |
|
||||
| `docs/plugins/homeassistant.md` | Home Assistant plugin: waiting updates, backup age, and installing updates after a confirmed backup |
|
||||
| `docs/plugins/syslog.md` | Syslog plugin: sources, patterns and alerts, publishing port 5514 (ingress or host mode under Swarm) |
|
||||
| `docs/plugins/prometheus-widgets.md` | Prometheus widget templates: writing, sharing and copying one in, every block, the format reference |
|
||||
| `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase |
|
||||
|
||||
@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
|
||||
|
||||
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
|
||||
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing. Height is judged with the window's highest reading left out, so one slow request can't carry a window past it, and a held level needs at least three readings behind its median, so a lone reading never opens an anomaly (`steward/core/anomalies.py`):
|
||||
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
|
||||
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
|
||||
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
|
||||
@@ -125,7 +125,7 @@ What learns a normal, and the floor each departure must clear (Settings → Thre
|
||||
|
||||
Host metrics learn with `parts=False`: the host learns, its cores (`<host>:core<N>`) don't. Disk space, a monitor that is down, an agent gone quiet and a router's 5xx share are failures and keep their fixed cutoffs (a router's shares are judged only over enough requests).
|
||||
|
||||
Pages grade by the normal through `anomalies.graded()` (hosts' `host_grades` and `worst_reading(normals=)`, `docker.service.usage(thresholds=, norms=)`, `monitors.service.latency_grade`): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages list their anomalies with `_anomaly_card.html`.
|
||||
Pages grade by the normal through `anomalies.graded()` (hosts' `host_grades` and `worst_reading(normals=)`, `docker.service.usage(thresholds=, norms=)`, `monitors.service.latency_grade`): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages each carry a one-line `_anomaly_card.html`. It says how many anomalies wait for review and names any that are out of their normal now. Its Review button opens Alerts → Anomalies narrowed to that resource (`?source=&resource=`), where they are reviewed.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+3
-1
@@ -76,6 +76,7 @@ Each role includes the tools of the roles above it in this list.
|
||||
| Monitors | `list_monitors`, `get_monitor` |
|
||||
| Alerts | `list_alerts`, `get_alert_history`, `list_alert_rules`, `get_alert_rule_options`, `list_maintenance_windows` |
|
||||
| Metric anomalies | `list_anomalies` (metrics that left their normal for the hour, to review first), `get_anomaly` (one anomaly with each occurrence against what is usual at that hour, the metric around it, what else happened on its host — alerts, Ansible runs, container deaths and starts, maintenance, reboots, other anomalies — and how earlier ones on the same metric were judged), `get_metric_normal` (the learned normal slot by slot, and the level that would be flagged), `get_anomaly_settings` (the review window and each floor) |
|
||||
| Thresholds | `get_thresholds` (the warn and crit cutoffs CPU, memory, disk, load and temperature are graded by on the pages) |
|
||||
| Automation | `list_playbooks`, `get_playbook`, `list_inventory`, `list_ansible_runs`, `get_ansible_run`, `wait_for_run`, `list_schedules` |
|
||||
| Audit | `list_audit_events` |
|
||||
| Timeline | `get_timeline` (everything that happened, newest first, in one list: alerts firing and clearing, playbook runs and what they changed, maintenance, reboots, deploys and crash-loops, reviewed anomalies, changes recorded by hand, changes to Steward's settings; by host, area or kind, and kept after the records it came from age out) |
|
||||
@@ -102,7 +103,8 @@ Each role includes the tools of the roles above it in this list.
|
||||
### Admin: destructive and remediation
|
||||
|
||||
`set_anomaly_settings` (the anomaly review window and each normal's floor, as
|
||||
Settings → Thresholds sets them), `set_alert_action` (the playbook a rule runs when it fires), `delete_monitor`,
|
||||
Settings → Thresholds sets them), `set_thresholds` (the warn and crit cutoffs host
|
||||
readings are graded by, as the same page sets them), `set_alert_action` (the playbook a rule runs when it fires), `delete_monitor`,
|
||||
`delete_alert_rule`, `prune_docker`, `run_maintenance` (update a host now, rebooting
|
||||
only if the update needs it, as the host page's Update now does).
|
||||
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Home Assistant plugin
|
||||
|
||||
Keeps a Home Assistant instance running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates, on a click or a weekly schedule, each only after a fresh backup is confirmed. It looks after Home Assistant itself and doesn't read your devices. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API.
|
||||
|
||||
## Setting it up
|
||||
|
||||
1. In Home Assistant, turn on automatic backups (Settings → System → Backups).
|
||||
2. Make a user for Steward (Settings → People → Add person), as an administrator if Steward should install updates.
|
||||
3. On Steward's Home Assistant page, choose Connect, enter Home Assistant's address, and continue to its sign-in page. Sign in as that user.
|
||||
|
||||
Your browser and Steward both need to reach Home Assistant at that address. Home Assistant sends your browser back to Steward's Public base URL (Settings → General), so open Steward at that address.
|
||||
|
||||
## What Steward holds
|
||||
|
||||
**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates. It is stored encrypted.
|
||||
|
||||
We recommend a service account, a user made just for Steward, rather than your own. What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.
|
||||
|
||||
Installing needs an administrator, because Home Assistant only lets administrators take backups. With any other user, Steward shows status only. Steward learns which it has by asking once after sign-in, and a user who isn't an administrator is refused that one request, which Home Assistant may show as a failed login notification.
|
||||
|
||||
Removing the connection ends the sign-in on Home Assistant. You can also end it yourself under Security on that user's profile, and Steward then stops asking until you sign in again. Home Assistant bans an address after a few refused requests, so Steward stops after one refusal until you sign in again or choose Try again.
|
||||
|
||||
## Installing an update
|
||||
|
||||
Install asks once, then Steward:
|
||||
|
||||
1. Asks for a backup and waits until Home Assistant's last successful backup time moves. If it doesn't, nothing is installed.
|
||||
2. Asks for the update and follows it until the new version is installed, or until a deadline: 20 minutes for an add-on, 30 for Core, 45 for the OS.
|
||||
|
||||
One runs at a time, and it carries on if Steward restarts. Firmware and custom integrations are left to Home Assistant, and Supervisor updates itself.
|
||||
|
||||
An OS update takes effect only after the machine reboots, and Install leaves that to you. Steward records the update as done once Home Assistant reports the new version, which it does before the reboot. Home Assistant then raises a repair asking for the reboot, which appears under System problems until you do it.
|
||||
|
||||
## System problems
|
||||
|
||||
Each read also asks Home Assistant whether the system itself has a problem, and the page and the overview show what it says:
|
||||
|
||||
- open repairs, from Settings → System → Repairs
|
||||
- the Supervisor calling the system unhealthy or unsupported
|
||||
- less than 10% of the disk free
|
||||
- starting in recovery mode, when its configuration didn't load
|
||||
- an add-on set to start on its own that isn't running
|
||||
|
||||
Repairs need an administrator's sign-in. Disk space and add-ons come from the Supervisor, so an install without one doesn't report them, and the page says so.
|
||||
|
||||
## Scheduled updates
|
||||
|
||||
Set a day and time under Scheduled updates on the page. It's off until you do. At that time Steward installs what's waiting, add-ons first, then Core, then the OS, one at a time and each after its own confirmed backup. It starts nothing new more than 4 hours after the slot. The first failure ends the run and the rest wait, and the page, the overview and `update_failed` say so. A Core update restarts Home Assistant with nobody there to watch, and an update can change how automations behave.
|
||||
|
||||
An OS update installed by the schedule also reboots the machine, with nobody there to watch. It counts as done once the new version is running. If that hasn't happened 20 minutes after the reboot, for example because the machine didn't come back or fell back to the old version, the update is marked failed.
|
||||
|
||||
## Metrics
|
||||
|
||||
Under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting`, `backup_age_hours`, `problems_open`, `disk_free_pct` and `update_failed`. Useful alert rules: `backup_age_hours` above 26 for a missed daily backup, `problems_open` above 0, and `update_failed` above 0. A rule's no-data warning covers Home Assistant going quiet.
|
||||
@@ -46,6 +46,8 @@ Each target also records four metrics about itself, under its own name:
|
||||
|
||||
A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.
|
||||
|
||||
**A refused token stops the scraping.** When an endpoint answers 401 or 403, Steward stops asking and says so on the target's page and in the overview, because asking again won't change the answer and some services, Home Assistant among them, ban an address after a few refusals. Saving the target, usually with a new token, or pausing and resuming it, starts it again. Any other failure is retried at the next scrape.
|
||||
|
||||
A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state.
|
||||
|
||||
**Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.
|
||||
|
||||
+48
-29
@@ -4,14 +4,16 @@
|
||||
anomalies and the alert rules and maintenance windows that name it.
|
||||
|
||||
The raw table holds millions of rows, too many to move in a migration that the
|
||||
app has to wait for. A small task moves a batch at a time in the background
|
||||
instead, one key after another, and does nothing once none are left. Network
|
||||
app has to wait for. A small task moves what it can in a short pass in the
|
||||
background instead, newest history first, and does nothing once none are left. Network
|
||||
rates stay where they are: they are kept per host, and age out with retention.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections import Counter
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import delete, select, text, update
|
||||
@@ -31,8 +33,10 @@ if TYPE_CHECKING:
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BATCH = 5000 # raw samples moved in one go
|
||||
BATCH = 1000 # raw samples moved in one go: each rewrites its row and every index on it
|
||||
EVERY_S = 15
|
||||
PASS_S = 2 # a key with only a few rows takes milliseconds, and there can be many
|
||||
REST = 4 # after a pass, wait this many times as long as it took, so ingest keeps the disk
|
||||
TRAFFIC = ("net_rx_bps", "net_tx_bps")
|
||||
_TASK_KEY = r"^[^/]+/.+\.([0-9]+|[a-z0-9]{25})\.[a-z0-9]{25}$"
|
||||
_NOT_TRAFFIC = "metric_name NOT IN ('net_rx_bps', 'net_tx_bps')"
|
||||
@@ -70,24 +74,27 @@ def service_key(old: str) -> str:
|
||||
|
||||
|
||||
async def old_keys(session) -> list[str]:
|
||||
"""Every task-named docker resource still holding something to move. The
|
||||
latest-value table stands for the raw samples, as a key stays in it until
|
||||
its samples have aged out."""
|
||||
found: set[str] = set()
|
||||
for model in (PluginMetricLatest, PluginMetricHourly):
|
||||
found.update((await session.execute(
|
||||
select(model.resource_name).distinct().where(
|
||||
model.source_module == "docker", model.metric_name.notin_(TRAFFIC),
|
||||
model.resource_name.op("~")(_TASK_KEY)))).scalars())
|
||||
for model in (MetricAnomaly, AlertRule):
|
||||
found.update((await session.execute(
|
||||
select(model.resource_name).distinct().where(
|
||||
model.source_module == "docker", model.resource_name.op("~")(_TASK_KEY)))).scalars())
|
||||
found.update((await session.execute(
|
||||
select(MaintenanceWindow.scope_resource).distinct().where(
|
||||
MaintenanceWindow.scope_source == "docker",
|
||||
MaintenanceWindow.scope_resource.op("~")(_TASK_KEY)))).scalars())
|
||||
return sorted(k for k in found if task_service(k.split("/", 1)[1]))
|
||||
"""Every task-named docker resource still holding something to move, to be
|
||||
taken from the end: those with raw samples last, as they are what a chart
|
||||
and a normal read first. The latest-value table stands for the raw samples,
|
||||
as a key stays in it until its samples have aged out."""
|
||||
async def names(column, *where) -> set[str]:
|
||||
found = (await session.execute(select(column).distinct().where(*where, column.op("~")(_TASK_KEY)))).scalars()
|
||||
return {k for k in found if task_service(k.split("/", 1)[1])}
|
||||
|
||||
recent = await names(PluginMetricLatest.resource_name, PluginMetricLatest.source_module == "docker",
|
||||
PluginMetricLatest.metric_name.notin_(TRAFFIC))
|
||||
hourly = await names(PluginMetricHourly.resource_name, PluginMetricHourly.source_module == "docker",
|
||||
PluginMetricHourly.metric_name.notin_(TRAFFIC))
|
||||
named = (await names(MetricAnomaly.resource_name, MetricAnomaly.source_module == "docker")
|
||||
| await names(AlertRule.resource_name, AlertRule.source_module == "docker")
|
||||
| await names(MaintenanceWindow.scope_resource, MaintenanceWindow.scope_source == "docker"))
|
||||
logger.info("Swarm history under task names: %d keys with raw samples, %d with hourly rows, "
|
||||
"%d named by anomalies, rules or windows", len(recent), len(hourly), len(named))
|
||||
by_service = Counter(task_service(k.split("/", 1)[1]) for k in hourly - recent)
|
||||
logger.info("Swarm history under task names, hourly only, most keys first: %s; for example %s",
|
||||
by_service.most_common(10), sorted(hourly - recent)[:3])
|
||||
return sorted(hourly - recent) + sorted((named - hourly) - recent) + sorted(recent)
|
||||
|
||||
|
||||
async def _merge_anomalies(session, new: str) -> None:
|
||||
@@ -148,9 +155,11 @@ async def move_key(session, old: str) -> bool:
|
||||
await session.execute(update(MaintenanceWindow).where(
|
||||
MaintenanceWindow.scope_source == "docker", MaintenanceWindow.scope_resource == old)
|
||||
.values(scope_resource=new))
|
||||
await session.execute(update(MetricAnomaly).where(
|
||||
renamed = await session.execute(update(MetricAnomaly).where(
|
||||
MetricAnomaly.source_module == "docker", MetricAnomaly.resource_name == old)
|
||||
.values(resource_name=new).execution_options(synchronize_session=False))
|
||||
if not renamed.rowcount:
|
||||
return True
|
||||
session.expire_all()
|
||||
await _merge_anomalies(session, new)
|
||||
await session.flush()
|
||||
@@ -163,17 +172,27 @@ async def move_key(session, old: str) -> bool:
|
||||
def make_rekey_task(app: "Quart") -> ScheduledTask:
|
||||
remaining: list[str] | None = None
|
||||
busy = asyncio.Lock()
|
||||
resume_at = 0.0
|
||||
|
||||
async def rekey() -> None:
|
||||
nonlocal remaining
|
||||
if remaining == [] or busy.locked():
|
||||
nonlocal remaining, resume_at
|
||||
started = time.monotonic()
|
||||
if remaining == [] or busy.locked() or started < resume_at:
|
||||
return
|
||||
async with busy, app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
if remaining is None:
|
||||
if remaining is None:
|
||||
async with session.begin():
|
||||
remaining = await old_keys(session)
|
||||
logger.info("Swarm history under task names: %d keys to move", len(remaining))
|
||||
if remaining and await move_key(session, remaining[-1]):
|
||||
remaining.pop()
|
||||
deadline = time.monotonic() + PASS_S
|
||||
while remaining and time.monotonic() < deadline:
|
||||
async with session.begin():
|
||||
finished = await move_key(session, remaining[-1])
|
||||
if not finished: # a full batch of samples is enough for one pass
|
||||
break
|
||||
remaining.pop()
|
||||
if not remaining:
|
||||
logger.info("Swarm history under task names: all moved")
|
||||
now = time.monotonic()
|
||||
resume_at = now + REST * (now - started)
|
||||
|
||||
return ScheduledTask(name="docker_rekey", coro_factory=rekey, interval_seconds=EVERY_S)
|
||||
|
||||
@@ -482,7 +482,7 @@ async def container_detail(host_id: str, name: str):
|
||||
resource = docker_svc.container_resource(host.name, name) if host is not None else None
|
||||
anomalies = await anom.views(db, await anom.list_anomalies(
|
||||
db, source="docker", resource=resource,
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=20),
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=100),
|
||||
links=False) if resource else []
|
||||
|
||||
if c is None:
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# plugins/homeassistant/__init__.py
|
||||
from __future__ import annotations
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from quart import Quart
|
||||
|
||||
_app: "Quart | None" = None
|
||||
|
||||
|
||||
def setup(app: "Quart") -> None:
|
||||
global _app
|
||||
_app = app
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate # noqa: F401 — registers the tables
|
||||
from .timeline import register
|
||||
register()
|
||||
|
||||
|
||||
def get_scheduled_tasks() -> list:
|
||||
from .scheduler import make_tick_task
|
||||
return [make_tick_task(_app)]
|
||||
|
||||
|
||||
def get_blueprint():
|
||||
from .routes import homeassistant_bp
|
||||
return homeassistant_bp
|
||||
|
||||
|
||||
def get_nav() -> list[dict]:
|
||||
return [{"label": "Home Assistant", "href": "/plugins/homeassistant/"}]
|
||||
|
||||
|
||||
def get_alert_metrics() -> dict[str, list[str]]:
|
||||
"""What the poll records, all under the resource "homeassistant"."""
|
||||
return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting", "problems_open", "disk_free_pct",
|
||||
"update_failed"]}
|
||||
|
||||
|
||||
def register_mcp_tools(register) -> None:
|
||||
from .mcp_tools import register_tools
|
||||
register_tools(register)
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
from .mcp_tools import mcp_overview as findings
|
||||
return await findings(db)
|
||||
@@ -0,0 +1,292 @@
|
||||
# plugins/homeassistant/client.py
|
||||
"""Talking to Home Assistant, with what its own sign-in gave Steward: a refresh
|
||||
token kept in the connection, and access tokens made from it that last half an
|
||||
hour. Repairs and the Supervisor's reports are only offered on its websocket. Home Assistant bans an address after a few refused requests, so nothing is
|
||||
sent without a token and a refusal stops the asking. Every failure comes back as
|
||||
one sentence a page can show."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import ssl
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from websockets.asyncio.client import connect as ws_connect
|
||||
from websockets.exceptions import WebSocketException
|
||||
|
||||
from steward.core.crypto import decrypt_secret
|
||||
|
||||
TIMEOUT_SECONDS = 15.0
|
||||
# Home Assistant answers an action when it has finished, minutes later for a
|
||||
# backup or an update, and carries on if the request ends. So its answer is only
|
||||
# waited for this long, and the outcome is read from its entities.
|
||||
ACTION_READ_SECONDS = 10.0
|
||||
MAX_BYTES = 16 * 1024 * 1024
|
||||
# An access token this close to running out isn't used for another request.
|
||||
EXPIRY_MARGIN_SECONDS = 60
|
||||
# One read of the system over the websocket, however its answers dribble in.
|
||||
SYSTEM_READ_SECONDS = 30
|
||||
|
||||
_access: dict[str, tuple[str, float]] = {} # refresh token -> access token, when to stop using it
|
||||
|
||||
|
||||
class Unreachable(Exception):
|
||||
"""Home Assistant didn't answer usefully; it may be restarting."""
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
"""Home Assistant refused Steward's sign-in, or has banned this address."""
|
||||
|
||||
|
||||
class Rejected(Exception):
|
||||
"""Home Assistant understood an action and wouldn't do it."""
|
||||
|
||||
|
||||
def _refusal(status: int) -> Refused | None:
|
||||
if status == 401:
|
||||
_access.clear()
|
||||
return Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.")
|
||||
if status == 403:
|
||||
return Refused("Home Assistant answered 403: it may have banned Steward's address "
|
||||
"after refused requests. Look in its ip_bans.yaml.")
|
||||
return None
|
||||
|
||||
|
||||
async def _request(method: str, url: str, path: str, verify_tls: bool, transport, *, token: str | None = None,
|
||||
read_timeout: float = TIMEOUT_SECONDS, **kwargs) -> tuple[int, bytes]:
|
||||
headers = {"Authorization": f"Bearer {token}"} if token else {}
|
||||
timeout = httpx.Timeout(TIMEOUT_SECONDS, read=read_timeout)
|
||||
try:
|
||||
# A slow drip of bytes can keep each read inside its own timeout for ever.
|
||||
async with asyncio.timeout(TIMEOUT_SECONDS + read_timeout):
|
||||
async with httpx.AsyncClient(timeout=timeout, verify=verify_tls, transport=transport,
|
||||
follow_redirects=False) as client:
|
||||
async with client.stream(method, url.rstrip("/") + path, headers=headers, **kwargs) as response:
|
||||
body = bytearray()
|
||||
async for chunk in response.aiter_bytes():
|
||||
body.extend(chunk)
|
||||
if len(body) > MAX_BYTES:
|
||||
raise Unreachable("Home Assistant's answer was too large to read.")
|
||||
return response.status_code, bytes(body)
|
||||
except TimeoutError:
|
||||
raise httpx.ReadTimeout("no complete answer in time") from None
|
||||
|
||||
|
||||
def _unreachable(exc: httpx.HTTPError) -> Unreachable:
|
||||
if isinstance(exc, httpx.TimeoutException):
|
||||
return Unreachable(f"Home Assistant didn't answer within {TIMEOUT_SECONDS:.0f} s.")
|
||||
return Unreachable(f"Couldn't reach Home Assistant: {exc}")
|
||||
|
||||
|
||||
# ── Signing in ───────────────────────────────────────────────────────────────
|
||||
|
||||
def sign_in_url(url: str, client_id: str, redirect_uri: str, state: str) -> str:
|
||||
"""Home Assistant's own sign-in page, which sends the browser back to
|
||||
`redirect_uri`. It accepts `client_id` as it is because the two share an
|
||||
address, so it needn't be able to reach Steward."""
|
||||
return f"{url.rstrip('/')}/auth/authorize?" + urlencode(
|
||||
{"response_type": "code", "client_id": client_id, "redirect_uri": redirect_uri, "state": state})
|
||||
|
||||
|
||||
async def _tokens(url: str, form: dict, verify_tls: bool, transport) -> dict:
|
||||
try:
|
||||
status, body = await _request("POST", url, "/auth/token", verify_tls, transport, data=form)
|
||||
except httpx.HTTPError as exc:
|
||||
raise _unreachable(exc) from None
|
||||
if status in (400, 401, 403):
|
||||
raise Refused("Home Assistant no longer accepts Steward's sign-in. Sign in again.")
|
||||
if status != 200:
|
||||
raise Unreachable(f"Home Assistant answered {status}.")
|
||||
try:
|
||||
found = json.loads(body)
|
||||
found["access_token"], int(found["expires_in"])
|
||||
except (ValueError, KeyError, TypeError):
|
||||
raise Unreachable("Home Assistant's answer wasn't a sign-in.") from None
|
||||
return found
|
||||
|
||||
|
||||
def _remember(refresh_token: str, found: dict) -> str:
|
||||
until = time.monotonic() + int(found["expires_in"]) - EXPIRY_MARGIN_SECONDS
|
||||
_access[refresh_token] = (found["access_token"], until)
|
||||
return found["access_token"]
|
||||
|
||||
|
||||
async def redeem_code(url: str, client_id: str, code: str, verify_tls: bool, transport=None) -> tuple[str, str]:
|
||||
"""(refresh token, access token) for the code the sign-in page sent back."""
|
||||
found = await _tokens(url, {"grant_type": "authorization_code", "client_id": client_id, "code": code},
|
||||
verify_tls, transport)
|
||||
refresh_token = found.get("refresh_token")
|
||||
if not refresh_token:
|
||||
raise Unreachable("Home Assistant's answer wasn't a sign-in.")
|
||||
return refresh_token, _remember(refresh_token, found)
|
||||
|
||||
|
||||
async def access_token(connection, transport=None) -> str:
|
||||
refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in")
|
||||
token, until = _access.get(refresh_token, (None, 0.0))
|
||||
if token and time.monotonic() < until:
|
||||
return token
|
||||
found = await _tokens(connection.url, {"grant_type": "refresh_token", "client_id": connection.client_id,
|
||||
"refresh_token": refresh_token}, connection.verify_tls, transport)
|
||||
return _remember(refresh_token, found)
|
||||
|
||||
|
||||
async def is_admin(url: str, token: str, verify_tls: bool, transport=None) -> bool:
|
||||
"""Whether the signed-in user is an administrator, which its API only says by
|
||||
refusing someone who isn't. So this is asked once, right after signing in, and
|
||||
a refusal here counts once toward a ban."""
|
||||
try:
|
||||
status, _ = await _request("POST", url, "/api/template", verify_tls, transport, token=token,
|
||||
json={"template": "1"})
|
||||
except httpx.HTTPError as exc:
|
||||
raise _unreachable(exc) from None
|
||||
if status == 401:
|
||||
return False
|
||||
if status != 200:
|
||||
raise (_refusal(status) or Unreachable(f"Home Assistant answered {status}."))
|
||||
return True
|
||||
|
||||
|
||||
async def revoke(connection, transport=None) -> None:
|
||||
"""End this sign-in on Home Assistant. It answers 200 whether or not it knew it."""
|
||||
refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in")
|
||||
_access.pop(refresh_token, None)
|
||||
try:
|
||||
status, _ = await _request("POST", connection.url, "/auth/revoke", connection.verify_tls, transport,
|
||||
data={"token": refresh_token})
|
||||
except httpx.HTTPError as exc:
|
||||
raise _unreachable(exc) from None
|
||||
if status != 200:
|
||||
raise Unreachable(f"Home Assistant answered {status}.")
|
||||
|
||||
|
||||
# ── Reading and acting ───────────────────────────────────────────────────────
|
||||
|
||||
async def _get(url: str, path: str, token: str, verify_tls: bool, transport) -> list | dict | None:
|
||||
"""The parsed JSON of a GET, or None for a 404."""
|
||||
try:
|
||||
status, body = await _request("GET", url, path, verify_tls, transport, token=token)
|
||||
except httpx.HTTPError as exc:
|
||||
raise _unreachable(exc) from None
|
||||
if refusal := _refusal(status):
|
||||
raise refusal
|
||||
if status == 404:
|
||||
return None
|
||||
if status != 200:
|
||||
raise Unreachable(f"Home Assistant answered {status}.")
|
||||
try:
|
||||
return json.loads(body)
|
||||
except ValueError:
|
||||
raise Unreachable("Home Assistant's answer wasn't JSON.") from None
|
||||
|
||||
|
||||
async def read_states(url: str, token: str, verify_tls: bool, transport=None) -> list[dict]:
|
||||
states = await _get(url, "/api/states", token, verify_tls, transport)
|
||||
if not isinstance(states, list):
|
||||
raise Unreachable("Home Assistant's answer wasn't a list of entities.")
|
||||
return states
|
||||
|
||||
|
||||
async def read_state(url: str, entity_id: str, token: str, verify_tls: bool, transport=None) -> dict | None:
|
||||
return await _get(url, f"/api/states/{entity_id}", token, verify_tls, transport)
|
||||
|
||||
|
||||
async def call_action(url: str, domain: str, action: str, data: dict, token: str, verify_tls: bool,
|
||||
transport=None) -> None:
|
||||
"""Ask Home Assistant to do something. Returning means it got the request,
|
||||
not that it has finished: a request cut off once sent has been taken up, and
|
||||
an update that restarts Home Assistant cuts its own request off."""
|
||||
try:
|
||||
status, body = await _request("POST", url, f"/api/services/{domain}/{action}", verify_tls, transport,
|
||||
token=token, read_timeout=ACTION_READ_SECONDS, json=data)
|
||||
except (httpx.ReadTimeout, httpx.ReadError, httpx.RemoteProtocolError):
|
||||
return
|
||||
except httpx.HTTPError as exc:
|
||||
raise _unreachable(exc) from None
|
||||
if refusal := _refusal(status):
|
||||
raise refusal
|
||||
if status != 200:
|
||||
try:
|
||||
said = json.loads(body).get("message")
|
||||
except (ValueError, AttributeError):
|
||||
said = None
|
||||
raise Rejected(said or f"Home Assistant answered {status}.")
|
||||
|
||||
|
||||
# ── The system, over the websocket ───────────────────────────────────────────
|
||||
|
||||
class _Socket:
|
||||
"""Home Assistant's websocket after sign-in: one question at a time, each with
|
||||
an id, and None for one it won't answer (no Supervisor, or not an administrator)."""
|
||||
|
||||
def __init__(self, ws):
|
||||
self.ws, self.asked = ws, 0
|
||||
|
||||
async def sign_in(self, token: str) -> None:
|
||||
await self.ws.recv()
|
||||
await self.ws.send(json.dumps({"type": "auth", "access_token": token}))
|
||||
if json.loads(await self.ws.recv()).get("type") != "auth_ok":
|
||||
raise Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.")
|
||||
|
||||
async def ask(self, **question):
|
||||
self.asked += 1
|
||||
await self.ws.send(json.dumps({"id": self.asked, **question}))
|
||||
while True:
|
||||
answer = json.loads(await self.ws.recv())
|
||||
if answer.get("id") == self.asked:
|
||||
return answer.get("result") if answer.get("success") else None
|
||||
|
||||
async def supervisor(self, endpoint: str):
|
||||
return await self.ask(type="supervisor/api", endpoint=endpoint, method="get")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _signed_in(url: str, token: str, verify_tls: bool, connect):
|
||||
"""A signed-in socket. `connect` stands in for the websocket in tests."""
|
||||
target = "ws" + url.rstrip("/").removeprefix("http") + "/api/websocket"
|
||||
options: dict = {"max_size": MAX_BYTES}
|
||||
if target.startswith("wss:") and not verify_tls:
|
||||
options["ssl"] = ssl.create_default_context()
|
||||
options["ssl"].check_hostname, options["ssl"].verify_mode = False, ssl.CERT_NONE
|
||||
try:
|
||||
async with asyncio.timeout(SYSTEM_READ_SECONDS):
|
||||
async with (connect or ws_connect)(target, **options) as ws:
|
||||
socket = _Socket(ws)
|
||||
await socket.sign_in(token)
|
||||
yield socket
|
||||
except TimeoutError:
|
||||
raise Unreachable(f"Home Assistant's system report didn't arrive within {SYSTEM_READ_SECONDS} s.") from None
|
||||
except (OSError, ValueError, KeyError, WebSocketException) as exc:
|
||||
raise Unreachable(f"Couldn't read Home Assistant's system report: {exc}") from None
|
||||
|
||||
|
||||
async def read_system(url: str, token: str, verify_tls: bool, connect=None) -> dict:
|
||||
"""What service.read_health needs, as Home Assistant answered it."""
|
||||
async with _signed_in(url, token, verify_tls, connect) as socket:
|
||||
found = {"config": await socket.ask(type="get_config"),
|
||||
"issues": (await socket.ask(type="repairs/list_issues") or {}).get("issues"),
|
||||
"resolution": await socket.supervisor("/resolution/info"),
|
||||
"host": await socket.supervisor("/host/info"), "titles": {}, "boots": {}}
|
||||
domains = sorted({i["domain"] for i in found["issues"] or []})
|
||||
if domains:
|
||||
titles = await socket.ask(type="frontend/get_translations", language="en",
|
||||
category="issues", integration=domains)
|
||||
found["titles"] = (titles or {}).get("resources") or {}
|
||||
found["addons"] = (await socket.supervisor("/addons") or {}).get("addons")
|
||||
# Whether one should be running is only on its own page, so a running one isn't asked.
|
||||
for addon in found["addons"] or []:
|
||||
if addon.get("state") != "started":
|
||||
info = await socket.supervisor(f"/addons/{addon['slug']}/info") or {}
|
||||
found["boots"][addon["slug"]] = info.get("boot")
|
||||
return found
|
||||
|
||||
|
||||
async def read_os(url: str, token: str, verify_tls: bool, connect=None) -> dict | None:
|
||||
"""The Supervisor's report on the OS: `version`, and `version_pending` while a
|
||||
written update waits for its reboot. None if it isn't answering yet, as just
|
||||
after a boot."""
|
||||
async with _signed_in(url, token, verify_tls, connect) as socket:
|
||||
return await socket.supervisor("/os/info")
|
||||
@@ -0,0 +1,168 @@
|
||||
"""The Home Assistant plugin's MCP tools and its get_overview findings.
|
||||
|
||||
get_home_assistant reads what the poll stored. update_home_assistant starts an
|
||||
update as the page's Install does. The sign-in Steward holds is never returned.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.core import weekly
|
||||
from steward.core.formatting import zone
|
||||
from steward.mcp._context import current_caller, mcp_app
|
||||
from steward.mcp.tools._audit import audit_mcp, mcp_username
|
||||
from steward.mcp.tools._shape import age_s, db_session, iso, now_utc, redact_url
|
||||
from steward.mcp.tools.overview import finding
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from . import schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .routes import connection_says
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
|
||||
from .updates import CannotStart, start_update
|
||||
|
||||
RECENT_UPDATES = 10
|
||||
|
||||
|
||||
def _schedule_view(plan, now) -> dict | None:
|
||||
if plan is None:
|
||||
return None
|
||||
return {"enabled": plan.enabled, "when": weekly.describe(plan),
|
||||
"next_slot_at": iso(weekly.next_slot(plan, now)) if plan.enabled else None,
|
||||
"last_slot_at": iso(weekly.fired_slot(plan)), "stopped": plan.stopped}
|
||||
|
||||
|
||||
def _run_view(r: HomeAssistantUpdate) -> dict:
|
||||
return {"update_id": r.id, "entity_id": r.entity_id, "title": r.title, "kind": r.kind,
|
||||
"from_version": r.from_version, "to_version": r.to_version, "state": r.state, "message": r.message,
|
||||
"started_by": r.started_by, "started_at": iso(r.started_at), "backup_at": iso(r.backup_at),
|
||||
"finished_at": iso(r.finished_at)}
|
||||
|
||||
|
||||
async def get_home_assistant() -> dict:
|
||||
"""Home Assistant as of the last poll: whether it answers, which updates are
|
||||
waiting, how old its last backup is, and the updates Steward was asked to install.
|
||||
|
||||
`installable` is what update_home_assistant takes (Core, OS, add-ons);
|
||||
`others_waiting` is firmware and custom integrations, which Steward leaves
|
||||
to Home Assistant. `can_install` is whether the Home Assistant user who signed
|
||||
Steward in is an administrator, which installing needs. `system.problems` is
|
||||
what's wrong with the system itself: open repairs, Supervisor health, low disk,
|
||||
recovery mode, add-ons that should be running. `schedule` is the weekly slot
|
||||
for installing what's waiting, null until one is set.
|
||||
"""
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
if c is None:
|
||||
return {"connected": False, "says": "No connection yet: an admin signs in on the Home Assistant page."}
|
||||
reach, says = connection_says(c, now)
|
||||
backup = c.backup or {}
|
||||
age = backup_age_hours(backup, now)
|
||||
return {"connected": True, "url": redact_url(c.url), "reach": reach, "says": says,
|
||||
"polled_at": iso(c.last_polled_at), "ok_age_s": age_s(c.last_ok_at, now), "can_install": c.can_install,
|
||||
"installable": [{k: u[k] for k in ("entity_id", "title", "kind", "installed", "latest", "in_progress")}
|
||||
for u in installable(c)],
|
||||
"others_waiting": [u["title"] for u in c.updates or [] if u["waiting"] and not u["kind"]],
|
||||
"backup": {"verdict": backup_verdict(backup, now)[1], "last_ok": iso(parse_moment(backup.get("last_ok"))),
|
||||
"age_hours": round(age, 1) if age is not None else None, "stale_after_hours": STALE_BACKUP_HOURS,
|
||||
"next": iso(parse_moment(backup.get("next"))), "last_result": backup.get("last_result")},
|
||||
"system": c.health or {}, "schedule": _schedule_view(plan, now),
|
||||
"updates": [_run_view(r) for r in runs]}
|
||||
|
||||
|
||||
async def update_home_assistant(entity_id: str) -> dict:
|
||||
"""Install one Core, OS or add-on update, as the page's Install does: Steward
|
||||
takes a backup, confirms it landed, installs, and follows it to its end.
|
||||
|
||||
entity_id: from get_home_assistant's `installable`.
|
||||
|
||||
An update restarts Home Assistant or the add-on. An OS update reads as done
|
||||
once it's written, and takes effect after a reboot, which Steward leaves to a
|
||||
person. Refused while another is under way, or if nothing is waiting there.
|
||||
Admin only, as on the page. Recorded in the audit log.
|
||||
"""
|
||||
caller = current_caller()
|
||||
try:
|
||||
run = await start_update(mcp_app(), entity_id, mcp_username(caller.username))
|
||||
except CannotStart as exc:
|
||||
raise ValueError(f"Nothing was started: {exc}.") from None
|
||||
await audit_mcp("homeassistant.update_started", "homeassistant_update", run.id,
|
||||
{"entity_id": entity_id, "title": run.title, "to": run.to_version})
|
||||
return {**_run_view(run), "next": "get_home_assistant follows it: `updates` shows its state."}
|
||||
|
||||
|
||||
async def set_home_assistant_schedule(enabled: bool, weekday: int = 5, at_time: str = "05:00",
|
||||
time_zone: str = "UTC") -> dict:
|
||||
"""Set the weekly slot for installing what's waiting, or turn it off, as the page does.
|
||||
|
||||
weekday: 0 for Monday to 6 for Sunday. at_time: "HH:MM" in time_zone.
|
||||
|
||||
At each slot Steward installs add-ons, then Core, then the OS, one at a time,
|
||||
each after a confirmed backup, stopping at the first failure. A Core update
|
||||
restarts Home Assistant and an OS update reboots the machine, which Steward
|
||||
does itself here, with nobody watching. The first slot is the next one from
|
||||
now. Admin only, and audited.
|
||||
"""
|
||||
try:
|
||||
weekly.parse_at(at_time)
|
||||
except ValueError:
|
||||
raise ValueError(f"at_time is HH:MM, not {at_time!r}.") from None
|
||||
if not 0 <= weekday <= 6:
|
||||
raise ValueError("weekday is 0 for Monday to 6 for Sunday.")
|
||||
if zone(time_zone) is timezone.utc and time_zone.upper() != "UTC":
|
||||
raise ValueError(f"Unknown time zone {time_zone!r}.")
|
||||
row = await schedule.save(mcp_app(), enabled=enabled, weekday=weekday, at_time=at_time, time_zone=time_zone)
|
||||
await audit_mcp("homeassistant.schedule_saved", "homeassistant_schedule", row.id,
|
||||
{"enabled": row.enabled, "when": weekly.describe(row)})
|
||||
return _schedule_view(row, now_utc())
|
||||
|
||||
|
||||
def overview_findings(c, latest_run, now, plan=None) -> list[dict]:
|
||||
"""A connection that isn't answering, a problem with the system, stale backups,
|
||||
a newest update that failed, and a scheduled run that stopped. Updates waiting
|
||||
aren't a problem to raise."""
|
||||
if c is None:
|
||||
return []
|
||||
found = []
|
||||
reach, says = connection_says(c, now)
|
||||
if reach == "down":
|
||||
found.append(finding("warning", "home_assistant_down", "Home Assistant",
|
||||
f"Home Assistant isn't answering: {says}", since=c.last_ok_at, next="get_home_assistant()"))
|
||||
problems = problems_open(c.health)
|
||||
if problems:
|
||||
more = f" (and {len(problems) - 1} more)" if len(problems) > 1 else ""
|
||||
found.append(finding("warning", "home_assistant_problem", "Home Assistant", f"{problems[0]['title']}{more}",
|
||||
next="get_home_assistant()"))
|
||||
tone, backup_says = backup_verdict(c.backup, now)
|
||||
if c.last_ok_at is not None and tone == "warn":
|
||||
found.append(finding("warning", "home_assistant_backup", "Home Assistant", backup_says,
|
||||
since=parse_moment((c.backup or {}).get("last_ok")), next="get_home_assistant()"))
|
||||
if latest_run is not None and latest_run.state == "failed":
|
||||
found.append(finding("warning", "home_assistant_update_failed", latest_run.title,
|
||||
f"The update of {latest_run.title} failed: {latest_run.message}",
|
||||
since=latest_run.finished_at, next="get_home_assistant()"))
|
||||
if plan is not None and plan.stopped:
|
||||
found.append(finding("warning", "home_assistant_schedule_stopped", "Home Assistant",
|
||||
f"The scheduled update run stopped: {plan.stopped}", since=weekly.fired_slot(plan),
|
||||
next="get_home_assistant()"))
|
||||
return found
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
latest = (await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
return overview_findings(c, latest, now_utc(), plan)
|
||||
|
||||
|
||||
def register_tools(register) -> None:
|
||||
register(get_home_assistant, role=UserRole.viewer, read_only=True)
|
||||
register(update_home_assistant, role=UserRole.admin, read_only=False)
|
||||
register(set_home_assistant_schedule, role=UserRole.admin, read_only=False)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Home Assistant: the connection and the updates Steward was asked to install
|
||||
|
||||
Revision ID: homeassistant_001_connection
|
||||
Revises: (none — a branch off core via depends_on)
|
||||
Create Date: 2026-10-31
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "homeassistant_001_connection"
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = "homeassistant"
|
||||
depends_on: Union[str, Sequence[str], None] = ("0004_app_settings",)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"homeassistant_connection",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("url", sa.String(500), nullable=False),
|
||||
sa.Column("token", sa.Text, nullable=False),
|
||||
sa.Column("webhook_id", sa.Text, nullable=False),
|
||||
sa.Column("verify_tls", sa.Boolean, nullable=False, server_default=sa.true()),
|
||||
sa.Column("refused", sa.Boolean, nullable=False, server_default=sa.false()),
|
||||
sa.Column("last_polled_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_ok_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_error", sa.Text, nullable=True),
|
||||
sa.Column("updates", sa.JSON, nullable=False, server_default="[]"),
|
||||
sa.Column("backup", sa.JSON, nullable=False, server_default="{}"),
|
||||
sa.Column("automation", sa.String(20), nullable=True),
|
||||
)
|
||||
op.create_table(
|
||||
"homeassistant_updates",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("entity_id", sa.String(255), nullable=False),
|
||||
sa.Column("title", sa.String(255), nullable=False),
|
||||
sa.Column("kind", sa.String(10), nullable=False),
|
||||
sa.Column("from_version", sa.String(100), nullable=True),
|
||||
sa.Column("to_version", sa.String(100), nullable=True),
|
||||
sa.Column("state", sa.String(20), nullable=False, server_default="backing_up"),
|
||||
sa.Column("message", sa.Text, nullable=True),
|
||||
sa.Column("started_by", sa.String(100), nullable=False, server_default=""),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("backup_before", sa.String(64), nullable=True),
|
||||
sa.Column("backup_event_before", sa.String(64), nullable=True),
|
||||
sa.Column("triggered_before", sa.String(64), nullable=True),
|
||||
sa.Column("backup_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("install_sent_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
op.execute("CREATE UNIQUE INDEX uq_homeassistant_update_under_way ON homeassistant_updates ((true)) "
|
||||
"WHERE state IN ('backing_up', 'installing')")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("homeassistant_updates")
|
||||
op.drop_table("homeassistant_connection")
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Home Assistant: Steward holds a sign-in, in place of a token and a webhook
|
||||
|
||||
Revision ID: homeassistant_002_sign_in
|
||||
Revises: homeassistant_001_connection
|
||||
Create Date: 2026-10-31
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "homeassistant_002_sign_in"
|
||||
down_revision: Union[str, None] = "homeassistant_001_connection"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# A saved token can't be refreshed like a sign-in, so that connection signs in again.
|
||||
op.execute("DELETE FROM homeassistant_connection")
|
||||
op.drop_column("homeassistant_connection", "webhook_id")
|
||||
op.drop_column("homeassistant_connection", "automation")
|
||||
op.drop_column("homeassistant_updates", "triggered_before")
|
||||
op.alter_column("homeassistant_connection", "token", new_column_name="refresh_token")
|
||||
op.add_column("homeassistant_connection", sa.Column("client_id", sa.String(500), nullable=False))
|
||||
op.add_column("homeassistant_connection",
|
||||
sa.Column("can_install", sa.Boolean, nullable=False, server_default=sa.false()))
|
||||
op.add_column("homeassistant_connection",
|
||||
sa.Column("signed_in_by", sa.String(100), nullable=False, server_default=""))
|
||||
op.add_column("homeassistant_connection", sa.Column("signed_in_at", sa.DateTime(timezone=True), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("DELETE FROM homeassistant_connection")
|
||||
op.drop_column("homeassistant_connection", "signed_in_at")
|
||||
op.drop_column("homeassistant_connection", "signed_in_by")
|
||||
op.drop_column("homeassistant_connection", "can_install")
|
||||
op.drop_column("homeassistant_connection", "client_id")
|
||||
op.alter_column("homeassistant_connection", "refresh_token", new_column_name="token")
|
||||
op.add_column("homeassistant_updates", sa.Column("triggered_before", sa.String(64), nullable=True))
|
||||
op.add_column("homeassistant_connection", sa.Column("automation", sa.String(20), nullable=True))
|
||||
op.add_column("homeassistant_connection", sa.Column("webhook_id", sa.Text, nullable=False))
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Home Assistant: what the poll reads of the system, and a weekly slot for updates
|
||||
|
||||
Revision ID: homeassistant_003_health
|
||||
Revises: homeassistant_002_sign_in
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "homeassistant_003_health"
|
||||
down_revision: Union[str, None] = "homeassistant_002_sign_in"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("homeassistant_connection",
|
||||
sa.Column("health", sa.JSON, nullable=False, server_default="{}"))
|
||||
op.create_table(
|
||||
"homeassistant_schedule",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("enabled", sa.Boolean, nullable=False),
|
||||
sa.Column("weekday", sa.Integer, nullable=False),
|
||||
sa.Column("at_time", sa.String(5), nullable=False),
|
||||
sa.Column("time_zone", sa.String(64), nullable=False),
|
||||
sa.Column("window_minutes", sa.Integer, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("last_slot_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("stopped", sa.Text, nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("homeassistant_schedule")
|
||||
op.drop_column("homeassistant_connection", "health")
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Home Assistant: an update the schedule installs can reboot the machine and follow it
|
||||
|
||||
Revision ID: homeassistant_004_reboot
|
||||
Revises: homeassistant_003_health
|
||||
Create Date: 2026-11-02
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "homeassistant_004_reboot"
|
||||
down_revision: Union[str, None] = "homeassistant_003_health"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _under_way(states: str) -> None:
|
||||
op.execute("DROP INDEX uq_homeassistant_update_under_way")
|
||||
op.execute("CREATE UNIQUE INDEX uq_homeassistant_update_under_way ON homeassistant_updates ((true)) "
|
||||
f"WHERE state IN ({states})")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("homeassistant_updates",
|
||||
sa.Column("reboot_after", sa.Boolean, nullable=False, server_default=sa.false()))
|
||||
op.add_column("homeassistant_updates", sa.Column("reboot_sent_at", sa.DateTime(timezone=True), nullable=True))
|
||||
_under_way("'backing_up', 'installing', 'rebooting'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
_under_way("'backing_up', 'installing'")
|
||||
op.drop_column("homeassistant_updates", "reboot_sent_at")
|
||||
op.drop_column("homeassistant_updates", "reboot_after")
|
||||
@@ -0,0 +1,92 @@
|
||||
# plugins/homeassistant/models.py
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Index, Integer, String, Text, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from steward.core.ids import time_ordered_id
|
||||
from steward.models.base import Base
|
||||
|
||||
|
||||
class HomeAssistantConnection(Base):
|
||||
"""The Home Assistant instance Steward looks after; at most one row.
|
||||
|
||||
`refresh_token` is what Home Assistant's sign-in gave Steward, encrypted. It
|
||||
has the same access as the user who signed in, whatever that is: Home
|
||||
Assistant has no narrower permission. `can_install` is whether that user was
|
||||
an administrator, which backups need. The last_* columns and `updates` and
|
||||
`backup` and `health` are the newest poll's outcome, which the pages read. `refused` stops
|
||||
all polling until someone signs in again or tries again: Home Assistant bans
|
||||
an address after a few refused requests.
|
||||
"""
|
||||
__tablename__ = "homeassistant_connection"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
|
||||
url: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
client_id: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
refresh_token: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
can_install: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
signed_in_by: Mapped[str] = mapped_column(String(100), nullable=False, default="")
|
||||
signed_in_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
verify_tls: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
refused: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
last_polled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_ok_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
updates: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
backup: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
health: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
|
||||
|
||||
class HomeAssistantSchedule(Base):
|
||||
"""The weekly slot for installing what's waiting; at most one row, none until an
|
||||
admin sets it. `last_slot_at` is the slot that last fired, and `stopped` says why
|
||||
that slot ended early. Times are read through steward.core.weekly."""
|
||||
__tablename__ = "homeassistant_schedule"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
weekday: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
at_time: Mapped[str] = mapped_column(String(5), nullable=False)
|
||||
time_zone: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
window_minutes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
|
||||
last_slot_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
stopped: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
|
||||
class HomeAssistantUpdate(Base):
|
||||
"""One update Steward was asked to install, from the click to its end. It
|
||||
moves through backing_up and installing to done or failed, and its row
|
||||
carries everything needed to carry on after Steward restarts. The *_before
|
||||
columns are Home Assistant's own readings at the start, so "a newer backup"
|
||||
is judged on its clock alone. One can be under way at a time. `reboot_after`
|
||||
sends it through rebooting too, for an OS update the schedule installs.
|
||||
"""
|
||||
__tablename__ = "homeassistant_updates"
|
||||
__table_args__ = (
|
||||
Index("uq_homeassistant_update_under_way", text("(true)"), unique=True,
|
||||
postgresql_where=text("state IN ('backing_up', 'installing', 'rebooting')")),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
|
||||
entity_id: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
title: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
kind: Mapped[str] = mapped_column(String(10), nullable=False)
|
||||
from_version: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
to_version: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
state: Mapped[str] = mapped_column(String(20), nullable=False, default="backing_up")
|
||||
message: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
started_by: Mapped[str] = mapped_column(String(100), nullable=False, default="")
|
||||
started_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
|
||||
backup_before: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
backup_event_before: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
backup_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
install_sent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
reboot_after: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
reboot_sent_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
@@ -0,0 +1,17 @@
|
||||
# plugins/homeassistant/plugin.yaml
|
||||
name: homeassistant
|
||||
version: "1.0.0"
|
||||
description: "Keeps Home Assistant running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates after a confirmed backup, on a click or a weekly schedule."
|
||||
author: "Steward"
|
||||
license: "MIT"
|
||||
min_app_version: "0.1.0"
|
||||
repository_url: "https://git.fabledsword.com/bvandeusen/FabledSteward"
|
||||
homepage: "https://git.fabledsword.com/bvandeusen/FabledSteward/src/branch/dev/plugins/homeassistant"
|
||||
tags:
|
||||
- home-assistant
|
||||
- updates
|
||||
- backups
|
||||
|
||||
# The connection is added and edited on the Home Assistant page
|
||||
# (/plugins/homeassistant/).
|
||||
config: {}
|
||||
@@ -0,0 +1,249 @@
|
||||
"""The Home Assistant page. Every GET reads Steward's own records, which the poll
|
||||
fills. Only signing in, removing the connection and Install, an admin's explicit
|
||||
actions, ask Home Assistant anything (System 178's ruling)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timezone
|
||||
from types import SimpleNamespace
|
||||
|
||||
from quart import Blueprint, current_app, redirect, render_template, request, session
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.core import weekly
|
||||
from steward.core.audit import log_audit
|
||||
from steward.core.crypto import encrypt_secret
|
||||
from steward.core.formatting import zone
|
||||
from steward.core.settings import public_base_url
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
from . import client, schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
|
||||
from .updates import UNDER_WAY, CannotStart, start_update
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
homeassistant_bp = Blueprint("homeassistant", __name__, template_folder="templates")
|
||||
|
||||
RECENT_UPDATES = 10
|
||||
_SIGN_IN = "homeassistant_sign_in" # in the session between leaving for Home Assistant and coming back
|
||||
|
||||
|
||||
def _actor() -> tuple[str | None, str]:
|
||||
return session.get("user_id"), session.get("username", "")
|
||||
|
||||
|
||||
async def _connection(db) -> HomeAssistantConnection | None:
|
||||
return (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
|
||||
|
||||
def connection_says(connection, now: datetime) -> tuple[str, str]:
|
||||
"""(badge state, sentence) for whether Home Assistant is answering."""
|
||||
if connection.refused:
|
||||
return "down", f"{connection.last_error} Steward has stopped asking until you sign in again or try again."
|
||||
if connection.last_polled_at is None:
|
||||
return "pending", "Not read yet: the first read runs within seconds."
|
||||
if connection.last_error:
|
||||
return "down", connection.last_error
|
||||
return "up", "Answering."
|
||||
|
||||
|
||||
async def _render_index(error: str | None = None, status: int = 200):
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
connection = await _connection(db)
|
||||
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
me = await db.get(User, session.get("user_id")) if session.get("user_id") else None
|
||||
if connection is None:
|
||||
return await render_template("homeassistant/index.html", connection=None, error=error), status
|
||||
default_zone = (me.time_zone if me and me.time_zone else "UTC")
|
||||
now = datetime.now(timezone.utc)
|
||||
reach, says = connection_says(connection, now)
|
||||
updates = connection.updates or []
|
||||
backup = connection.backup or {}
|
||||
tone, backup_says = backup_verdict(backup, now)
|
||||
return await render_template(
|
||||
"homeassistant/index.html", connection=connection, error=error, reach=reach, says=says,
|
||||
installable=installable(connection),
|
||||
others=[u for u in updates if u["waiting"] and not u["kind"]],
|
||||
current=[u for u in updates if not u["waiting"] and u["kind"]],
|
||||
runs=runs, under_way=next((r for r in runs if r.state in UNDER_WAY), None),
|
||||
backup=backup, backup_tone=tone, backup_says=backup_says,
|
||||
backup_at=parse_moment(backup.get("last_ok")), backup_next=parse_moment(backup.get("next")),
|
||||
backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS,
|
||||
health=connection.health or {}, problems=problems_open(connection.health),
|
||||
plan=plan, plan_when=weekly.describe(plan) if plan else None,
|
||||
plan_next=weekly.next_slot(plan, now) if plan and plan.enabled else None,
|
||||
weekdays=weekly.WEEKDAYS, default_zone=default_zone), status
|
||||
|
||||
|
||||
@homeassistant_bp.get("/")
|
||||
@require_role(UserRole.viewer)
|
||||
async def index():
|
||||
return await _render_index()
|
||||
|
||||
|
||||
@homeassistant_bp.post("/install")
|
||||
@require_role(UserRole.admin)
|
||||
async def install():
|
||||
form = await request.form
|
||||
entity_id = (form.get("entity_id") or "").strip()
|
||||
user_id, username = _actor()
|
||||
try:
|
||||
run = await start_update(current_app, entity_id, username)
|
||||
except CannotStart as exc:
|
||||
return await _render_index(f"Nothing was started: {exc}.", 409)
|
||||
await log_audit(current_app, user_id, username, "homeassistant.update_started",
|
||||
entity_type="homeassistant_update", entity_id=run.id,
|
||||
detail={"entity_id": entity_id, "title": run.title, "to": run.to_version})
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
@homeassistant_bp.post("/schedule")
|
||||
@require_role(UserRole.admin)
|
||||
async def save_schedule():
|
||||
"""Set the weekly slot for installing what's waiting, or turn it off."""
|
||||
form = await request.form
|
||||
time_zone = (form.get("time_zone") or "UTC").strip()
|
||||
try:
|
||||
weekday, at_time = int(form.get("weekday", "")), (form.get("at_time") or "").strip()
|
||||
weekly.parse_at(at_time)
|
||||
except ValueError:
|
||||
return await _render_index("Choose a day and a time.", 422)
|
||||
if not 0 <= weekday <= 6 or (zone(time_zone) is timezone.utc and time_zone.upper() != "UTC"):
|
||||
return await _render_index(f"Choose a day, and a known time zone rather than “{time_zone}”.", 422)
|
||||
row = await schedule.save(current_app, enabled=bool(form.get("enabled")), weekday=weekday,
|
||||
at_time=at_time, time_zone=time_zone)
|
||||
await log_audit(current_app, *_actor(), "homeassistant.schedule_saved", entity_type="homeassistant_schedule",
|
||||
entity_id=row.id, detail={"enabled": row.enabled, "when": weekly.describe(row)})
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
# ── The connection ───────────────────────────────────────────────────────────
|
||||
|
||||
async def _render_form(connection, url: str | None = None, verify_tls: bool | None = None,
|
||||
error: str | None = None, status: int = 200):
|
||||
return await render_template(
|
||||
"homeassistant/form.html", connection=connection, error=error,
|
||||
url=url if url is not None else (connection.url if connection else ""),
|
||||
verify_tls=verify_tls if verify_tls is not None else (connection.verify_tls if connection else True)), status
|
||||
|
||||
|
||||
@homeassistant_bp.get("/connection")
|
||||
@require_role(UserRole.admin)
|
||||
async def connection_form():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
connection = await _connection(db)
|
||||
return await _render_form(connection)
|
||||
|
||||
|
||||
@homeassistant_bp.post("/connection/sign-in")
|
||||
@require_role(UserRole.admin)
|
||||
async def sign_in():
|
||||
"""Send the browser to Home Assistant's own sign-in page."""
|
||||
form = await request.form
|
||||
url = (form.get("url") or "").strip().rstrip("/")
|
||||
verify_tls = bool(form.get("verify_tls"))
|
||||
if not url.startswith(("http://", "https://")):
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
connection = await _connection(db)
|
||||
return await _render_form(connection, url, verify_tls, "Enter Home Assistant's address: http://… or https://…", 422)
|
||||
base, state = public_base_url(request), secrets.token_urlsafe(32)
|
||||
client_id = f"{base}/"
|
||||
session[_SIGN_IN] = {"state": state, "url": url, "verify_tls": verify_tls, "client_id": client_id}
|
||||
return redirect(client.sign_in_url(url, client_id, f"{base}/plugins/homeassistant/connection/callback", state))
|
||||
|
||||
|
||||
@homeassistant_bp.get("/connection/callback")
|
||||
@require_role(UserRole.admin)
|
||||
async def sign_in_callback():
|
||||
"""Where Home Assistant sends the browser back. A GET that asks Home Assistant
|
||||
something, because the admin's sign-in has only just happened and its code
|
||||
lasts minutes. The state shows the sign-in began here, for this admin."""
|
||||
pending = session.pop(_SIGN_IN, None)
|
||||
state, code = request.args.get("state", ""), request.args.get("code", "")
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
connection = await _connection(db)
|
||||
if not pending or not hmac.compare_digest(pending["state"].encode(), state.encode()):
|
||||
return await _render_form(connection, error="That sign-in didn't start from this page, so it wasn't used. "
|
||||
"If it keeps happening, check Public base URL in Settings → General: "
|
||||
"Steward must be opened at that address.", status=400)
|
||||
url, verify_tls, client_id = pending["url"], pending["verify_tls"], pending["client_id"]
|
||||
if not code:
|
||||
return await _render_form(connection, url, verify_tls, "Home Assistant didn't finish the sign-in. Nothing changed.", 400)
|
||||
try:
|
||||
refresh_token, token = await client.redeem_code(url, client_id, code, verify_tls)
|
||||
can_install = await client.is_admin(url, token, verify_tls)
|
||||
except (client.Refused, client.Unreachable) as exc:
|
||||
return await _render_form(connection, url, verify_tls, f"The sign-in couldn't be finished: {exc}", 502)
|
||||
user_id, username = _actor()
|
||||
previous = None
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
connection = await _connection(db)
|
||||
if connection is None:
|
||||
connection = HomeAssistantConnection(url=url, client_id=client_id, refresh_token="")
|
||||
db.add(connection)
|
||||
else:
|
||||
previous = (connection.url, connection.client_id, connection.refresh_token, connection.verify_tls)
|
||||
connection.url, connection.client_id, connection.verify_tls = url, client_id, verify_tls
|
||||
connection.refresh_token = encrypt_secret(refresh_token)
|
||||
connection.can_install, connection.signed_in_by = can_install, username
|
||||
connection.signed_in_at = datetime.now(timezone.utc)
|
||||
connection.refused, connection.last_error, connection.last_polled_at = False, None, None
|
||||
await db.flush()
|
||||
connection_id = connection.id
|
||||
if previous is not None:
|
||||
await _end_sign_in(SimpleNamespace(url=previous[0], client_id=previous[1], refresh_token=previous[2],
|
||||
verify_tls=previous[3]))
|
||||
await log_audit(current_app, user_id, username, "homeassistant.connection_saved",
|
||||
entity_type="homeassistant_connection", entity_id=connection_id,
|
||||
detail={"name": url, "can_install": can_install})
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
async def _end_sign_in(connection) -> bool:
|
||||
"""End a sign-in Steward no longer needs, so it doesn't stay valid on Home
|
||||
Assistant. It can be ended there by hand too, so failing here stops nothing."""
|
||||
try:
|
||||
await client.revoke(connection)
|
||||
except (client.Refused, client.Unreachable) as exc:
|
||||
logger.warning("Home Assistant: couldn't end Steward's old sign-in: %s", exc)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@homeassistant_bp.post("/connection/retry")
|
||||
@require_role(UserRole.admin)
|
||||
async def retry_connection():
|
||||
"""Ask again after a refusal, once a ban has been dealt with."""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
connection = await _connection(db)
|
||||
if connection is not None:
|
||||
connection.refused, connection.last_polled_at = False, None
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
@homeassistant_bp.post("/connection/delete")
|
||||
@require_role(UserRole.admin)
|
||||
async def delete_connection():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
connection = await _connection(db)
|
||||
if connection is None:
|
||||
return redirect("/plugins/homeassistant/")
|
||||
connection_id, url = connection.id, connection.url
|
||||
await db.delete(connection)
|
||||
ended = await _end_sign_in(connection)
|
||||
await log_audit(current_app, *_actor(), "homeassistant.connection_removed",
|
||||
entity_type="homeassistant_connection", entity_id=connection_id, detail={"name": url, "sign_in_ended": ended})
|
||||
if not ended:
|
||||
return await _render_index("Removed from Steward, but Home Assistant couldn't be reached to end the sign-in. "
|
||||
"End it yourself under Security on that user's profile in Home Assistant.")
|
||||
return redirect("/plugins/homeassistant/")
|
||||
@@ -0,0 +1,95 @@
|
||||
# plugins/homeassistant/schedule.py
|
||||
"""Installing what's waiting at a weekly slot: one at a time through the same path
|
||||
as the page's Install, each after its own confirmed backup. Add-ons go first and
|
||||
the OS last, as it reboots the machine. The first failure ends the slot, leaving the
|
||||
rest for someone to look at. Everything is on the rows, so a restart of Steward or
|
||||
of Home Assistant only delays the next step."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.core import weekly
|
||||
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import installable
|
||||
from .updates import CannotStart, start_update, under_way
|
||||
|
||||
ACTOR = "schedule"
|
||||
WINDOW_MINUTES = 240
|
||||
_ORDER = {"addon": 0, "core": 1, "os": 2}
|
||||
|
||||
|
||||
def next_to_install(waiting: list[dict], tried: set[str]) -> dict | None:
|
||||
left = [u for u in waiting if u["entity_id"] not in tried and not u["in_progress"]]
|
||||
return min(left, key=lambda u: _ORDER[u["kind"]], default=None)
|
||||
|
||||
|
||||
async def _stop(app, schedule_id: str, why: str) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantSchedule, schedule_id)
|
||||
if row is not None:
|
||||
row.stopped = why
|
||||
|
||||
|
||||
async def _fire(app, schedule_id: str, slot: datetime) -> None:
|
||||
"""Open the slot, and have the next poll say what's waiting at this moment."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantSchedule, schedule_id)
|
||||
row.last_slot_at, row.stopped = slot, None
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is not None:
|
||||
connection.last_polled_at = None
|
||||
|
||||
|
||||
async def step(app, transport=None) -> None:
|
||||
"""Fire a slot that has come, or start the next update of one that's open."""
|
||||
now = datetime.now(timezone.utc)
|
||||
async with app.db_sessionmaker() as session:
|
||||
schedule = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if schedule is None or connection is None:
|
||||
return
|
||||
slot = weekly.is_due(schedule, now)
|
||||
if slot is not None:
|
||||
await _fire(app, schedule.id, slot)
|
||||
return
|
||||
slot = weekly.fired_slot(schedule)
|
||||
if not schedule.enabled or schedule.stopped or slot is None \
|
||||
or now - slot >= timedelta(minutes=schedule.window_minutes):
|
||||
return
|
||||
# Only on a fresh, good read: Home Assistant may be restarting from the last update.
|
||||
if connection.refused or connection.last_polled_at is None or connection.last_error:
|
||||
return
|
||||
if await under_way(session) is not None:
|
||||
return
|
||||
runs = (await session.execute(select(HomeAssistantUpdate).where(
|
||||
HomeAssistantUpdate.started_at >= slot))).scalars().all()
|
||||
failed = next((r for r in runs if r.state == "failed"), None)
|
||||
if failed is not None:
|
||||
await _stop(app, schedule.id, f"{failed.title}: {failed.message} Nothing more was installed in this slot.")
|
||||
return
|
||||
update = next_to_install(installable(connection), {r.entity_id for r in runs})
|
||||
if update is None:
|
||||
return
|
||||
try:
|
||||
await start_update(app, update["entity_id"], ACTOR, transport, reboot_after=update["kind"] == "os")
|
||||
except CannotStart as exc:
|
||||
await _stop(app, schedule.id, f"{update['title']} wasn't started: {exc}. Nothing more was installed in this slot.")
|
||||
|
||||
|
||||
async def save(app, *, enabled: bool, weekday: int, at_time: str, time_zone: str) -> HomeAssistantSchedule:
|
||||
"""Set the slot. Its first is the next one from now, not one already past."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
if row is None:
|
||||
row = HomeAssistantSchedule(window_minutes=WINDOW_MINUTES)
|
||||
session.add(row)
|
||||
row.enabled, row.weekday, row.time_zone = enabled, weekday, time_zone
|
||||
row.at_time = f"{weekly.parse_at(at_time):%H:%M}"
|
||||
row.last_slot_at, row.stopped = datetime.now(timezone.utc), None
|
||||
return row
|
||||
@@ -0,0 +1,116 @@
|
||||
# plugins/homeassistant/scheduler.py
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.core.scheduler import ScheduledTask
|
||||
|
||||
from . import client, schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_health, read_snapshot
|
||||
from .updates import advance
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from quart import Quart
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# An update under way is followed each tick, with one small read of its entity.
|
||||
TICK_SECONDS = 15
|
||||
# The whole entity list is a few hundred KB, and updates and backups change rarely.
|
||||
POLL_SECONDS = 300
|
||||
# A step past this is given up until the next tick, whatever the client's own
|
||||
# timeouts did: a slow drip of bytes can keep each read inside its timeout for ever.
|
||||
STEP_DEADLINE_SECONDS = 60
|
||||
|
||||
|
||||
def make_tick_task(app: "Quart") -> ScheduledTask:
|
||||
busy = asyncio.Lock()
|
||||
|
||||
async def tick() -> None:
|
||||
if busy.locked():
|
||||
return
|
||||
async with busy:
|
||||
for step in (poll, advance, schedule.step):
|
||||
try:
|
||||
await asyncio.wait_for(step(app), STEP_DEADLINE_SECONDS)
|
||||
except TimeoutError:
|
||||
logger.warning("Home Assistant %s gave up after %d s", step.__name__, STEP_DEADLINE_SECONDS)
|
||||
|
||||
return ScheduledTask(name="homeassistant_tick", coro_factory=tick,
|
||||
interval_seconds=TICK_SECONDS, run_on_startup=True)
|
||||
|
||||
|
||||
def is_due(connection, now: datetime) -> bool:
|
||||
"""Whether to poll. Never after a refusal: another refused request counts
|
||||
toward Home Assistant banning this address."""
|
||||
if connection.refused:
|
||||
return False
|
||||
if connection.last_polled_at is None:
|
||||
return True
|
||||
return (now - connection.last_polled_at).total_seconds() >= POLL_SECONDS - TICK_SECONDS / 2
|
||||
|
||||
|
||||
async def update_failed(session) -> bool:
|
||||
"""Whether the newest update failed, or the schedule couldn't start one."""
|
||||
stopped = (await session.execute(select(HomeAssistantSchedule.stopped))).scalars().first()
|
||||
newest = (await session.execute(select(HomeAssistantUpdate.state).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
|
||||
return bool(stopped) or newest == "failed"
|
||||
|
||||
|
||||
async def poll(app: "Quart", transport=None, connect=None) -> None:
|
||||
from steward.core.alerts import MetricBatch
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
async with app.db_sessionmaker() as session:
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is None or not is_due(connection, now):
|
||||
return
|
||||
snapshot, health, error, refused = None, None, None, False
|
||||
try:
|
||||
token = await client.access_token(connection, transport)
|
||||
snapshot = read_snapshot(await client.read_states(connection.url, token, connection.verify_tls, transport))
|
||||
try:
|
||||
health = read_health(await client.read_system(connection.url, token, connection.verify_tls, connect))
|
||||
except client.Unreachable as exc:
|
||||
# Updates and backups were read, so it's answering. The last report stands, and says it's old.
|
||||
health = {**(connection.health or {}), "error": str(exc)}
|
||||
except client.Refused as exc:
|
||||
snapshot, error, refused = None, str(exc), True
|
||||
except client.Unreachable as exc:
|
||||
error = str(exc)
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantConnection, connection.id)
|
||||
if row is None: # removed while it was being polled
|
||||
return
|
||||
batch = MetricBatch()
|
||||
row.last_polled_at, row.last_error, row.refused = now, error, refused
|
||||
if snapshot is None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=0.0, recorded_at=now)
|
||||
else:
|
||||
row.last_ok_at = now
|
||||
row.updates, row.backup, row.health = snapshot["updates"], snapshot["backup"], health
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=1.0, recorded_at=now)
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="updates_waiting",
|
||||
value=len(installable(row)), recorded_at=now)
|
||||
age = backup_age_hours(snapshot["backup"], now)
|
||||
if age is not None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="backup_age_hours",
|
||||
value=age, recorded_at=now)
|
||||
if not health.get("error"):
|
||||
if health["repairs_read"]:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="problems_open",
|
||||
value=len(health["problems"]), recorded_at=now)
|
||||
if health["disk_free_pct"] is not None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="disk_free_pct",
|
||||
value=health["disk_free_pct"], recorded_at=now)
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="update_failed",
|
||||
value=1.0 if await update_failed(session) else 0.0, recorded_at=now)
|
||||
await batch.record(session)
|
||||
@@ -0,0 +1,161 @@
|
||||
# plugins/homeassistant/service.py
|
||||
"""Reading what Home Assistant reports: which updates it knows of, which of them
|
||||
Steward may install, how its backups stand, and whether the system has a problem."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SOURCE = "homeassistant"
|
||||
RESOURCE = "homeassistant"
|
||||
# Backups run daily, so past a day and a bit one has been missed.
|
||||
STALE_BACKUP_HOURS = 26
|
||||
LOW_DISK_PCT = 10
|
||||
|
||||
CORE = "update.home_assistant_core_update"
|
||||
OS = "update.home_assistant_operating_system_update"
|
||||
SUPERVISOR = "update.home_assistant_supervisor_update"
|
||||
LAST_BACKUP = "sensor.backup_last_successful_automatic_backup"
|
||||
NEXT_BACKUP = "sensor.backup_next_scheduled_automatic_backup"
|
||||
BACKUP_MANAGER = "sensor.backup_backup_manager_state"
|
||||
BACKUP_EVENT = "event.backup_automatic_backup"
|
||||
|
||||
_KIND_WORDS = {"core": "Core", "os": "Operating system", "addon": "Add-on"}
|
||||
_KIND_ORDER = {"core": 0, "os": 1, "addon": 2, None: 3}
|
||||
|
||||
def classify(entity: dict) -> str | None:
|
||||
""""core", "os" or "addon" for an update Steward may install; None for the
|
||||
rest: Supervisor updates itself, and firmware and custom integrations are
|
||||
Home Assistant's own to offer."""
|
||||
entity_id = entity.get("entity_id")
|
||||
if entity_id == CORE:
|
||||
return "core"
|
||||
if entity_id == OS:
|
||||
return "os"
|
||||
if entity_id == SUPERVISOR:
|
||||
return None
|
||||
picture = (entity.get("attributes") or {}).get("entity_picture") or ""
|
||||
return "addon" if picture.startswith("/api/hassio/addons/") else None
|
||||
|
||||
|
||||
def _update(entity: dict) -> dict:
|
||||
a = entity.get("attributes") or {}
|
||||
kind = classify(entity)
|
||||
url = a.get("release_url") or ""
|
||||
return {"entity_id": entity["entity_id"], "kind": kind, "kind_word": _KIND_WORDS.get(kind, "Other"),
|
||||
"title": a.get("title") or a.get("friendly_name") or entity["entity_id"],
|
||||
"installed": a.get("installed_version"), "latest": a.get("latest_version"),
|
||||
"waiting": entity.get("state") == "on", "in_progress": bool(a.get("in_progress")),
|
||||
"auto_update": bool(a.get("auto_update")),
|
||||
"release_url": url if url.startswith(("http://", "https://")) else None}
|
||||
|
||||
|
||||
def _state(entity: dict | None) -> str | None:
|
||||
value = (entity or {}).get("state")
|
||||
return None if value in (None, "unknown", "unavailable") else value
|
||||
|
||||
|
||||
def parse_moment(value: str | None) -> datetime | None:
|
||||
try:
|
||||
moment = datetime.fromisoformat(value) if value else None
|
||||
except ValueError:
|
||||
return None
|
||||
if moment is not None and moment.tzinfo is None:
|
||||
moment = moment.replace(tzinfo=timezone.utc)
|
||||
return moment
|
||||
|
||||
|
||||
def is_newer(value: str | None, than: str | None) -> bool:
|
||||
"""Whether reading `value` is a later moment than `than`, where `than` may be never."""
|
||||
moment, before = parse_moment(value), parse_moment(than)
|
||||
return moment is not None and (before is None or moment > before)
|
||||
|
||||
|
||||
def read_snapshot(states: list[dict]) -> dict:
|
||||
"""What a poll keeps of Home Assistant's entities."""
|
||||
by_id = {e.get("entity_id"): e for e in states}
|
||||
updates = [_update(e) for i, e in by_id.items() if (i or "").startswith("update.") and e.get("state") != "unavailable"]
|
||||
updates.sort(key=lambda u: (not u["waiting"], _KIND_ORDER[u["kind"]], u["title"].lower()))
|
||||
event = by_id.get(BACKUP_EVENT) or {}
|
||||
event_attrs = event.get("attributes") or {}
|
||||
return {
|
||||
"updates": updates,
|
||||
"backup": {"last_ok": _state(by_id.get(LAST_BACKUP)), "next": _state(by_id.get(NEXT_BACKUP)),
|
||||
"manager": _state(by_id.get(BACKUP_MANAGER)), "last_result": event_attrs.get("event_type"),
|
||||
"failed_reason": event_attrs.get("failed_reason"), "known": LAST_BACKUP in by_id},
|
||||
}
|
||||
|
||||
|
||||
def backup_age_hours(backup: dict | None, now: datetime) -> float | None:
|
||||
last = parse_moment((backup or {}).get("last_ok"))
|
||||
return None if last is None else max(0.0, (now - last).total_seconds() / 3600)
|
||||
|
||||
|
||||
def installable(connection) -> list[dict]:
|
||||
"""The waiting updates Steward may install."""
|
||||
return [u for u in (connection.updates or []) if u["waiting"] and u["kind"]]
|
||||
|
||||
|
||||
def backup_verdict(backup: dict | None, now: datetime) -> tuple[str, str]:
|
||||
"""(tone, sentence) for how the backups stand."""
|
||||
backup = backup or {}
|
||||
if not backup.get("known"):
|
||||
return "warn", "Home Assistant reports no automatic backups. Turn them on under Settings → System → Backups."
|
||||
age = backup_age_hours(backup, now)
|
||||
if age is None:
|
||||
return "warn", "Home Assistant has no successful automatic backup yet."
|
||||
if backup.get("last_result") == "failed":
|
||||
reason = backup.get("failed_reason")
|
||||
return "warn", f"The newest backup attempt failed{f' ({reason})' if reason else ''}."
|
||||
if age > STALE_BACKUP_HOURS:
|
||||
return "warn", f"The last good backup is {age:.0f} hours old, past the {STALE_BACKUP_HOURS} hours expected."
|
||||
return "ok", "Backups are up to date."
|
||||
|
||||
|
||||
def _issue_title(issue: dict, titles: dict) -> str:
|
||||
key = issue.get("translation_key") or issue.get("issue_id") or ""
|
||||
title = titles.get(f"component.{issue.get('domain')}.issues.{key}.title")
|
||||
if not title:
|
||||
return key.removeprefix("issue_").replace("_", " ").capitalize() or "A repair is open"
|
||||
for name, value in (issue.get("translation_placeholders") or {}).items():
|
||||
title = title.replace("{" + name + "}", str(value))
|
||||
return title
|
||||
|
||||
|
||||
def read_health(found: dict) -> dict:
|
||||
"""What a poll keeps of client.read_system: the problems to show, and the disk.
|
||||
A part Home Assistant wouldn't answer is left out rather than read as fine:
|
||||
`repairs_read` says whether the repairs were seen, and `supervised` whether a
|
||||
Supervisor answered, as it doesn't on every install or for every user."""
|
||||
problems = []
|
||||
if (found.get("config") or {}).get("recovery_mode"):
|
||||
problems.append({"kind": "recovery", "severity": "critical",
|
||||
"title": "Home Assistant started in recovery mode: its configuration didn't load."})
|
||||
issues = found.get("issues")
|
||||
for issue in issues or []:
|
||||
if not issue.get("ignored") and not issue.get("dismissed_version"):
|
||||
problems.append({"kind": "repair", "severity": issue.get("severity") or "warning",
|
||||
"title": _issue_title(issue, found.get("titles") or {})})
|
||||
resolution = found.get("resolution") or {}
|
||||
for word in ("unhealthy", "unsupported"):
|
||||
for reason in resolution.get(word) or []:
|
||||
problems.append({"kind": "supervisor", "severity": "error" if word == "unhealthy" else "warning",
|
||||
"title": f"The Supervisor calls this system {word}: {str(reason).replace('_', ' ')}."})
|
||||
host = found.get("host") or {}
|
||||
free, used, total = host.get("disk_free"), host.get("disk_used"), host.get("disk_total")
|
||||
free_pct = round(100 * free / total, 1) if free is not None and total else None
|
||||
# Used and free don't add up to the total: the system reserves some of the disk.
|
||||
used_pct = round(100 * used / total, 1) if used is not None and total else None
|
||||
if free_pct is not None and free_pct < LOW_DISK_PCT:
|
||||
problems.append({"kind": "disk", "severity": "error", "title": f"Only {free:g} GB of {total:g} GB free."})
|
||||
boots = found.get("boots") or {}
|
||||
for addon in found.get("addons") or []:
|
||||
if boots.get(addon.get("slug")) == "auto":
|
||||
problems.append({"kind": "addon", "severity": "error",
|
||||
"title": f"{addon.get('name') or addon.get('slug')} should start on its own and isn't running."})
|
||||
return {"repairs_read": issues is not None, "supervised": found.get("resolution") is not None, "problems": problems,
|
||||
"disk_free_gb": free, "disk_used_gb": used, "disk_total_gb": total,
|
||||
"disk_free_pct": free_pct, "disk_used_pct": used_pct}
|
||||
|
||||
|
||||
def problems_open(health: dict | None) -> list[dict]:
|
||||
return (health or {}).get("problems") or []
|
||||
@@ -0,0 +1,50 @@
|
||||
{# plugins/homeassistant/templates/homeassistant/form.html — where Home Assistant
|
||||
is, what signing in gives Steward, and leaving for its sign-in page. #}
|
||||
{% extends "base.html" %}
|
||||
{% from "_macros.html" import crumbs, page_header %}
|
||||
{% block title %}Home Assistant connection — Steward{% endblock %}
|
||||
{% block breadcrumb %}{{ crumbs([("Home Assistant", "/plugins/homeassistant/"), ("Connection", "")]) }}{% endblock %}
|
||||
{% block content %}
|
||||
{{ page_header("Home Assistant connection") }}
|
||||
|
||||
<div class="form-page">
|
||||
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
|
||||
{% if connection %}
|
||||
<p class="field-hint">Signed in{% if connection.signed_in_at %} {{ connection.signed_in_at|ago }}{% endif %}{% if connection.signed_in_by %} by {{ connection.signed_in_by }}{% endif %}, as {{ "an administrator: Steward can install updates" if connection.can_install else "a user who isn't an administrator: Steward shows status only" }}.</p>
|
||||
{% endif %}
|
||||
<form method="post" action="/plugins/homeassistant/connection/sign-in">{{ csrf_field() }}
|
||||
<section class="card form-section" aria-labelledby="ha-sign-in">
|
||||
<h2 id="ha-sign-in" class="section-title">Sign in with Home Assistant</h2>
|
||||
<div class="field">
|
||||
<label class="field-label" for="ha-url">Address</label>
|
||||
<input class="input mono" id="ha-url" name="url" required maxlength="500" value="{{ url or '' }}" placeholder="http://homeassistant.local:8123" autocomplete="off">
|
||||
<span class="field-hint">Both your browser and Steward need to reach it at this address.</span>
|
||||
</div>
|
||||
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.9rem;">
|
||||
<input type="checkbox" name="verify_tls" value="on"{% if verify_tls %} checked{% endif %}> Check its HTTPS certificate
|
||||
</label>
|
||||
<p class="tone-warn"><strong>Steward stores a credential with the same access as the user who signs in.</strong> Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates.</p>
|
||||
<p class="field-hint"><strong>We recommend a service account:</strong> a Home Assistant user made just for Steward (Settings → People → Add person). What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.</p>
|
||||
<p class="field-hint" style="margin-bottom:0;">Installing updates needs an administrator. With any other user, Steward shows status only.</p>
|
||||
</section>
|
||||
|
||||
<div class="form-actions">
|
||||
<button type="submit" class="btn">{{ "Sign in again" if connection else "Continue to Home Assistant" }}</button>
|
||||
<a href="/plugins/homeassistant/" class="btn btn-ghost">Cancel</a>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
{% if connection %}
|
||||
<section class="card danger-zone" aria-labelledby="ha-remove">
|
||||
<div>
|
||||
<h2 id="ha-remove" class="section-title">Remove this connection</h2>
|
||||
<p class="field-hint">Steward stops reading Home Assistant, forgets its credential and ends the sign-in there.</p>
|
||||
</div>
|
||||
<form method="post" action="/plugins/homeassistant/connection/delete"
|
||||
data-confirm="Remove the Home Assistant connection? Steward stops reading it and forgets its credential.">{{ csrf_field() }}
|
||||
<button class="btn btn-danger" type="submit">Remove connection</button>
|
||||
</form>
|
||||
</section>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,150 @@
|
||||
{# plugins/homeassistant/templates/homeassistant/index.html — whether Home
|
||||
Assistant answers, which updates are waiting with Install beside each, how its
|
||||
backups stand, and the updates Steward has run. Refreshes itself while one is
|
||||
under way. #}
|
||||
{% extends "base.html" %}
|
||||
{% from "_macros.html" import page_header, status_badge, stat_tile, empty_state %}
|
||||
{% block title %}Home Assistant — Steward{% endblock %}
|
||||
{% block content %}
|
||||
{% set admin = session.user_role == 'admin' %}
|
||||
{% call page_header("Home Assistant", connection.url if connection else none) %}
|
||||
{% if admin and connection %}<a class="btn btn-sm btn-ghost" href="/plugins/homeassistant/connection">Connection</a>{% endif %}
|
||||
{% endcall %}
|
||||
|
||||
{% if not connection %}
|
||||
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
|
||||
<div class="card">{{ empty_state("Not connected. Sign in to a Home Assistant instance to see its waiting updates and backups, and to install updates after a confirmed backup.",
|
||||
"Connect" if admin else none, "/plugins/homeassistant/connection") }}</div>
|
||||
{% else %}
|
||||
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
|
||||
<div id="ha-body"{% if under_way %} hx-get="/plugins/homeassistant/" hx-select="#ha-body" hx-trigger="every 15s" hx-swap="outerHTML"{% endif %}>
|
||||
<p class="page-verdict">
|
||||
{% if under_way %}{{ status_badge('pending', 'Updating') }}
|
||||
{{ under_way.title }}: {{ {'backing_up': "taking a backup first", 'rebooting': "rebooting into " ~ under_way.to_version}.get(under_way.state, "installing " ~ under_way.to_version) }}. Home Assistant may stop answering for a while.
|
||||
{% elif reach == 'up' %}{{ status_badge('up', 'Answering') }}
|
||||
{% if installable %}{{ installable|length }} update{{ "s" if installable|length != 1 }} waiting.{% else %}Everything Steward can update is up to date.{% endif %}
|
||||
Read {{ connection.last_polled_at|ago }}.
|
||||
{% elif reach == 'down' %}{{ status_badge('down', 'Not answering') }} <span class="tone-crit">{{ says }}</span>
|
||||
{% if connection.last_ok_at %}Last answered {{ connection.last_ok_at|ago }}; what's below is from then.{% endif %}
|
||||
{% if admin and connection.refused %}<form method="post" action="/plugins/homeassistant/connection/retry" style="display:inline;">{{ csrf_field() }}<button class="btn btn-sm btn-ghost" type="submit">Try again</button></form>{% endif %}
|
||||
{% else %}{{ status_badge('pending', 'Not read yet') }} {{ says }}{% endif %}
|
||||
</p>
|
||||
|
||||
{% if not connection.can_install %}
|
||||
<p class="tone-warn">The Home Assistant user who signed in isn't an administrator, so updates can't be installed from here.
|
||||
{% if admin %}<a href="/plugins/homeassistant/connection">Sign in again</a> as an administrator to install them.{% endif %}</p>
|
||||
{% endif %}
|
||||
|
||||
<div class="stat-strip">
|
||||
{{ stat_tile("System problems", problems|length if health.repairs_read else "—", state="degraded" if problems else none) }}
|
||||
{{ stat_tile("Storage", ("%s of %s GB (%.0f%%)"|format(health.disk_used_gb, health.disk_total_gb, health.disk_used_pct)) if health.disk_used_pct is number else "—") }}
|
||||
{{ stat_tile("Updates waiting", installable|length if connection.last_ok_at else "—") }}
|
||||
{{ stat_tile("Last good backup", (backup_at|ago) if backup_at else "—", state="degraded" if backup_tone == 'warn' else none) }}
|
||||
{{ stat_tile("Next backup", (backup_next|ago) if backup_next else "—") }}
|
||||
</div>
|
||||
<p class="{{ 'tone-warn' if backup_tone == 'warn' else 'field-hint' }}">{{ backup_says }}</p>
|
||||
|
||||
<section class="card" aria-labelledby="ha-system">
|
||||
<h2 id="ha-system" class="section-title">System</h2>
|
||||
{% if problems %}
|
||||
<table class="table">
|
||||
<thead><tr><th scope="col">Problem</th><th scope="col">Severity</th></tr></thead>
|
||||
<tbody>
|
||||
{% for p in problems %}<tr><td>{{ p.title }}</td><td>{{ status_badge('down' if p.severity in ('critical', 'error') else 'degraded', p.severity|capitalize) }}</td></tr>{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="field-hint">Fix these in Home Assistant: Settings → System → Repairs.</p>
|
||||
{% elif health.repairs_read %}
|
||||
<p class="field-hint">No repairs open, and nothing wrong reported.</p>
|
||||
{% elif connection.last_ok_at %}
|
||||
<p class="tone-warn">Home Assistant wouldn't share its repairs with this user. {% if admin %}<a href="/plugins/homeassistant/connection">Sign in again</a> as an administrator to see them.{% endif %}</p>
|
||||
{% endif %}
|
||||
{% if health.repairs_read and not health.supervised %}<p class="field-hint">No Supervisor answered, so disk space and add-ons aren't checked here.</p>{% endif %}
|
||||
{% if health.error %}<p class="tone-warn" style="margin-bottom:0;">This is from an earlier read. {{ health.error }}</p>{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="card" aria-labelledby="ha-updates">
|
||||
<h2 id="ha-updates" class="section-title">Updates</h2>
|
||||
{% if installable %}
|
||||
<table class="table">
|
||||
<thead><tr><th scope="col">What</th><th scope="col">Now</th><th scope="col">New</th>{% if admin %}<th scope="col"></th>{% endif %}</tr></thead>
|
||||
<tbody>
|
||||
{% for u in installable %}
|
||||
<tr>
|
||||
<td>{{ u.title }}<div class="cell-sub">{{ u.kind_word }}{% if u.auto_update %}, updates itself{% endif %}</div></td>
|
||||
<td class="mono">{{ u.installed or "—" }}</td>
|
||||
<td class="mono">{% if u.release_url %}<a href="{{ u.release_url }}" rel="noopener noreferrer" target="_blank">{{ u.latest or "—" }}</a>{% else %}{{ u.latest or "—" }}{% endif %}</td>
|
||||
{% if admin %}<td style="text-align:right;">
|
||||
{% if connection.can_install and not under_way and not connection.refused and not u.in_progress %}
|
||||
<form method="post" action="/plugins/homeassistant/install"
|
||||
data-confirm="Install {{ u.title }} {{ u.latest }}? Steward takes a backup first and won't install without it. {{ 'Home Assistant restarts.' if u.kind == 'core' else ('It takes effect once you reboot the machine, which Steward leaves to you.' if u.kind == 'os' else 'The add-on restarts.') }}">{{ csrf_field() }}
|
||||
<input type="hidden" name="entity_id" value="{{ u.entity_id }}">
|
||||
<button class="btn btn-sm" type="submit">Install</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</td>{% endif %}
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<p class="field-hint" style="margin:0;">Nothing waiting.</p>
|
||||
{% endif %}
|
||||
{% if current %}<p class="field-hint">Up to date: {% for u in current %}{{ u.title }} {{ u.installed }}{{ ", " if not loop.last }}{% endfor %}.</p>{% endif %}
|
||||
{% if others %}<p class="field-hint" style="margin-bottom:0;">Waiting, for Home Assistant to install: {% for u in others %}{{ u.title }}{{ ", " if not loop.last }}{% endfor %}. Steward installs Core, OS and add-on updates only.</p>{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="card" aria-labelledby="ha-schedule">
|
||||
<h2 id="ha-schedule" class="section-title">Scheduled updates</h2>
|
||||
{% if plan and plan.enabled %}
|
||||
<p class="field-hint">{{ plan_when }}. Next: {{ plan_next|ago }}. Add-ons first, then Core, then the OS and its reboot, each after a confirmed backup. A failure ends the run and the rest wait.</p>
|
||||
{% else %}
|
||||
<p class="field-hint">Off. Updates are only installed when someone chooses Install.</p>
|
||||
{% endif %}
|
||||
{% if plan and plan.stopped %}<p class="tone-warn">The last run stopped early. {{ plan.stopped }}</p>{% endif %}
|
||||
{% if admin %}
|
||||
<p class="tone-warn">A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch. An update can also change how automations behave.</p>
|
||||
<form method="post" action="/plugins/homeassistant/schedule" style="display:flex;flex-wrap:wrap;gap:0.75rem;align-items:flex-end;">{{ csrf_field() }}
|
||||
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.4rem;">
|
||||
<input type="checkbox" name="enabled" value="on"{% if plan and plan.enabled %} checked{% endif %}> Install on a schedule
|
||||
</label>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-day">Day</label>
|
||||
<select class="input" id="ha-day" name="weekday">{% for d in weekdays %}<option value="{{ loop.index0 }}"{% if loop.index0 == (plan.weekday if plan else 5) %} selected{% endif %}>{{ d }}</option>{% endfor %}</select>
|
||||
</div>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-time">Time</label>
|
||||
<input class="input" id="ha-time" type="time" name="at_time" required value="{{ plan.at_time if plan else '05:00' }}">
|
||||
</div>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-zone">Time zone</label>
|
||||
<input class="input" id="ha-zone" name="time_zone" required value="{{ plan.time_zone if plan else default_zone }}">
|
||||
</div>
|
||||
<button class="btn btn-sm" type="submit">Save</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
{% if runs %}
|
||||
<section class="card" aria-labelledby="ha-runs">
|
||||
<h2 id="ha-runs" class="section-title">Updates run from Steward</h2>
|
||||
<table class="table">
|
||||
<thead><tr><th scope="col">What</th><th scope="col">Versions</th><th scope="col">Outcome</th><th scope="col">Started</th></tr></thead>
|
||||
<tbody>
|
||||
{% for r in runs %}
|
||||
<tr>
|
||||
<td>{{ r.title }}<div class="cell-sub">by {{ r.started_by or "Steward" }}</div></td>
|
||||
<td class="mono">{{ r.from_version or "—" }} → {{ r.to_version or "—" }}</td>
|
||||
<td>{% if r.state == 'done' %}{{ status_badge('up', 'Done') }}
|
||||
{% elif r.state == 'failed' %}{{ status_badge('down', 'Failed') }}<div class="cell-sub">{{ r.message }}</div>
|
||||
{% else %}{{ status_badge('pending', {'backing_up': 'Taking a backup', 'rebooting': 'Rebooting'}.get(r.state, 'Installing')) }}{% endif %}</td>
|
||||
<td>{{ r.started_at|ago }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,34 @@
|
||||
# plugins/homeassistant/timeline.py
|
||||
"""What Home Assistant maintenance puts on the timeline: each update once it ends."""
|
||||
from __future__ import annotations
|
||||
|
||||
from steward.timeline import catalog
|
||||
from steward.timeline.record import record_or_update
|
||||
|
||||
TYPES = (
|
||||
("homeassistant.updated", "Home Assistant updated", "automation", "service", True,
|
||||
"Steward installed a Home Assistant update after a confirmed backup."),
|
||||
("homeassistant.update_failed", "Home Assistant update failed", "automation", "service", True,
|
||||
"A Home Assistant update Steward was asked to install didn't finish."),
|
||||
)
|
||||
|
||||
|
||||
def register() -> None:
|
||||
for t in TYPES:
|
||||
catalog.register_type(*t)
|
||||
from .models import HomeAssistantUpdate
|
||||
catalog.register_ref("homeassistant_update", HomeAssistantUpdate)
|
||||
|
||||
|
||||
async def update_entry(session, run) -> None:
|
||||
done = run.state == "done"
|
||||
versions = f"{run.from_version} to {run.to_version}" if run.to_version else "a new version"
|
||||
summary = (f"{run.title} updated from {versions}" if done
|
||||
else f"{run.title} update failed: {run.message}")
|
||||
await record_or_update(
|
||||
session, ("homeassistant_update", run.id),
|
||||
type="homeassistant.updated" if done else "homeassistant.update_failed",
|
||||
at=run.started_at, until=run.finished_at, summary=summary, subject=run.title,
|
||||
detail={"entity_id": run.entity_id, "from": run.from_version, "to": run.to_version,
|
||||
"url": "/plugins/homeassistant/"},
|
||||
actor=run.started_by or "Steward")
|
||||
@@ -0,0 +1,259 @@
|
||||
# plugins/homeassistant/updates.py
|
||||
"""Installing one update: take a backup, confirm it landed, install, and follow
|
||||
it to its end. Its row holds where it has got to, so Steward restarting, or Home
|
||||
Assistant restarting as an update does, only delays the next step."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from . import client
|
||||
from .models import HomeAssistantConnection, HomeAssistantUpdate
|
||||
from .service import BACKUP_EVENT, LAST_BACKUP, is_newer, parse_moment
|
||||
from .timeline import update_entry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
UNDER_WAY = ("backing_up", "installing", "rebooting")
|
||||
BACKUP_DEADLINE_S = 30 * 60
|
||||
REBOOT_DEADLINE_S = 20 * 60
|
||||
# An OS update downloads and writes a whole system image, which is slow on a Pi. Home
|
||||
# Assistant reports the new version as soon as that's written, before the reboot that
|
||||
# makes it run, and raises a repair asking for that reboot. The schedule's own OS
|
||||
# updates make that reboot; one started by a person leaves it to them.
|
||||
INSTALL_DEADLINE_S = {"addon": 20 * 60, "core": 30 * 60, "os": 45 * 60}
|
||||
|
||||
|
||||
class CannotStart(Exception):
|
||||
"""Why an update wasn't started, said to the person."""
|
||||
|
||||
|
||||
async def _mark_refused(app, connection_id: str, said: str) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantConnection, connection_id)
|
||||
if row is not None:
|
||||
row.refused, row.last_error = True, said
|
||||
|
||||
|
||||
async def under_way(session) -> HomeAssistantUpdate | None:
|
||||
return (await session.execute(select(HomeAssistantUpdate).where(
|
||||
HomeAssistantUpdate.state.in_(UNDER_WAY)))).scalars().first()
|
||||
|
||||
|
||||
async def start_update(app, entity_id: str, username: str, transport=None, *, reboot_after: bool = False) -> HomeAssistantUpdate:
|
||||
"""Start installing `entity_id`, an explicit action that may contact Home
|
||||
Assistant. Raises CannotStart, with nothing started, unless the update is
|
||||
still waiting and the signed-in user may install it. `reboot_after` reboots
|
||||
the machine once it's written, and it's done when the new version is running."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is None:
|
||||
raise CannotStart("Home Assistant isn't connected")
|
||||
if connection.refused:
|
||||
raise CannotStart("Home Assistant refused Steward's sign-in")
|
||||
if await under_way(session) is not None:
|
||||
raise CannotStart("another update is under way")
|
||||
if not connection.can_install:
|
||||
raise CannotStart("the Home Assistant user who signed in isn't an administrator")
|
||||
url, verify = connection.url, connection.verify_tls
|
||||
known = next((u for u in connection.updates or [] if u["entity_id"] == entity_id), None)
|
||||
if known is None or not known["kind"]:
|
||||
raise CannotStart("that isn't a Core, OS or add-on update")
|
||||
try:
|
||||
token = await client.access_token(connection, transport)
|
||||
current = await client.read_state(url, entity_id, token, verify, transport)
|
||||
backup = await client.read_state(url, LAST_BACKUP, token, verify, transport)
|
||||
event = await client.read_state(url, BACKUP_EVENT, token, verify, transport)
|
||||
except client.Refused as exc:
|
||||
await _mark_refused(app, connection.id, str(exc))
|
||||
raise CannotStart(str(exc).rstrip(".")) from None
|
||||
except client.Unreachable as exc:
|
||||
raise CannotStart(str(exc).rstrip(".")) from None
|
||||
if backup is None:
|
||||
raise CannotStart("Home Assistant has no automatic backups to confirm one with")
|
||||
if current is None or current.get("state") != "on":
|
||||
raise CannotStart("that update is no longer waiting")
|
||||
attrs = current.get("attributes") or {}
|
||||
run = HomeAssistantUpdate(
|
||||
entity_id=entity_id, title=known["title"], kind=known["kind"],
|
||||
from_version=attrs.get("installed_version"), to_version=attrs.get("latest_version"),
|
||||
started_by=username, backup_before=backup.get("state"),
|
||||
backup_event_before=(event or {}).get("state"), reboot_after=reboot_after)
|
||||
async with app.db_sessionmaker() as session:
|
||||
try:
|
||||
async with session.begin():
|
||||
session.add(run)
|
||||
await session.flush()
|
||||
run_id = run.id
|
||||
except IntegrityError:
|
||||
raise CannotStart("another update is under way") from None
|
||||
try:
|
||||
await client.call_action(url, "backup", "create_automatic", {}, token, verify, transport)
|
||||
except client.Refused as exc:
|
||||
await _mark_refused(app, connection.id, str(exc))
|
||||
await _finish(app, run_id, "failed", f"The backup couldn't be started: {exc} Nothing was installed.")
|
||||
raise CannotStart(str(exc).rstrip(".")) from None
|
||||
except (client.Unreachable, client.Rejected) as exc:
|
||||
await _finish(app, run_id, "failed", f"The backup couldn't be started: {exc} Nothing was installed.")
|
||||
raise CannotStart(str(exc).rstrip(".")) from None
|
||||
async with app.db_sessionmaker() as session:
|
||||
return await session.get(HomeAssistantUpdate, run_id)
|
||||
|
||||
|
||||
async def _finish(app, run_id: str, state: str, message: str | None) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
run = await session.get(HomeAssistantUpdate, run_id)
|
||||
run.state, run.message, run.finished_at = state, message, datetime.now(timezone.utc)
|
||||
# The page should show the new versions now, not at the next poll.
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is not None:
|
||||
connection.last_polled_at = None
|
||||
try:
|
||||
async with session.begin_nested():
|
||||
await update_entry(session, run)
|
||||
except Exception:
|
||||
logger.exception("Home Assistant update %s: its timeline entry could not be written", run_id)
|
||||
|
||||
|
||||
async def advance(app, transport=None, connect=None) -> None:
|
||||
"""Move the update under way one step on, if there is one."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
run = await under_way(session)
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if run is None:
|
||||
return
|
||||
if connection is None:
|
||||
await _finish(app, run.id, "failed", "The Home Assistant connection was removed.")
|
||||
return
|
||||
try:
|
||||
if run.state == "backing_up":
|
||||
await _advance_backup(app, run, connection, transport)
|
||||
elif run.state == "installing":
|
||||
await _advance_install(app, run, connection, transport)
|
||||
else:
|
||||
await _advance_reboot(app, run, connection, transport, connect)
|
||||
except client.Refused as exc:
|
||||
await _mark_refused(app, connection.id, str(exc))
|
||||
await _finish(app, run.id, "failed", str(exc))
|
||||
except client.Rejected as exc:
|
||||
await _finish(app, run.id, "failed", f"Home Assistant wouldn't install it: {exc}")
|
||||
except client.Unreachable as exc:
|
||||
# Expected while Home Assistant restarts: the next tick asks again.
|
||||
logger.info("Home Assistant update %s: %s", run.id, exc)
|
||||
await _past_deadline(app, run)
|
||||
|
||||
|
||||
def _elapsed(since: datetime) -> float:
|
||||
return (datetime.now(timezone.utc) - since).total_seconds()
|
||||
|
||||
|
||||
async def _past_deadline(app, run) -> bool:
|
||||
if run.state == "backing_up" and _elapsed(run.started_at) > BACKUP_DEADLINE_S:
|
||||
await _finish(app, run.id, "failed",
|
||||
f"No new backup within {BACKUP_DEADLINE_S // 60} minutes. Nothing was installed.")
|
||||
return True
|
||||
if run.state == "installing":
|
||||
limit = INSTALL_DEADLINE_S.get(run.kind, INSTALL_DEADLINE_S["os"])
|
||||
if _elapsed(run.install_sent_at or run.backup_at or run.started_at) > limit:
|
||||
await _finish(app, run.id, "failed",
|
||||
f"Still not on {run.to_version} after {limit // 60} minutes. Look in Home Assistant.")
|
||||
return True
|
||||
if run.state == "rebooting" and _elapsed(run.reboot_sent_at or run.install_sent_at or run.started_at) > REBOOT_DEADLINE_S:
|
||||
await _finish(app, run.id, "failed", f"{run.to_version} was installed, but isn't running "
|
||||
f"{REBOOT_DEADLINE_S // 60} minutes after the reboot. Look in Home Assistant.")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _advance_backup(app, run, connection, transport) -> None:
|
||||
token = await client.access_token(connection, transport)
|
||||
url, verify = connection.url, connection.verify_tls
|
||||
backup = await client.read_state(url, LAST_BACKUP, token, verify, transport)
|
||||
value = (backup or {}).get("state")
|
||||
if is_newer(value, run.backup_before):
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantUpdate, run.id)
|
||||
row.state, row.backup_at = "installing", parse_moment(value)
|
||||
return
|
||||
event = await client.read_state(url, BACKUP_EVENT, token, verify, transport)
|
||||
if event and event.get("state") != run.backup_event_before \
|
||||
and (event.get("attributes") or {}).get("event_type") == "failed":
|
||||
reason = (event.get("attributes") or {}).get("failed_reason")
|
||||
await _finish(app, run.id, "failed",
|
||||
f"The backup failed{f' ({reason})' if reason else ''}. Nothing was installed.")
|
||||
return
|
||||
await _past_deadline(app, run)
|
||||
|
||||
|
||||
async def _mark_sent(app, run_id: str, at: datetime | None) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantUpdate, run_id)
|
||||
row.install_sent_at = at
|
||||
|
||||
|
||||
async def _advance_install(app, run, connection, transport) -> None:
|
||||
token = await client.access_token(connection, transport)
|
||||
url, verify = connection.url, connection.verify_tls
|
||||
if run.install_sent_at is None:
|
||||
# Marked first: Steward stopping here leaves an install that runs out its
|
||||
# deadline, where asking twice would fail one that is going fine.
|
||||
await _mark_sent(app, run.id, datetime.now(timezone.utc))
|
||||
try:
|
||||
await client.call_action(url, "update", "install", {"entity_id": run.entity_id}, token, verify, transport)
|
||||
except client.Unreachable:
|
||||
await _mark_sent(app, run.id, None)
|
||||
raise
|
||||
return
|
||||
current = await client.read_state(url, run.entity_id, token, verify, transport)
|
||||
attrs = (current or {}).get("attributes") or {}
|
||||
installed = attrs.get("installed_version")
|
||||
if installed and (installed == run.to_version
|
||||
or (current.get("state") == "off" and installed != run.from_version)):
|
||||
if run.reboot_after:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantUpdate, run.id)
|
||||
row.state = "rebooting"
|
||||
return
|
||||
await _finish(app, run.id, "done", None)
|
||||
return
|
||||
await _past_deadline(app, run)
|
||||
|
||||
|
||||
async def _mark_reboot_sent(app, run_id: str, at: datetime | None) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantUpdate, run_id)
|
||||
row.reboot_sent_at = at
|
||||
|
||||
|
||||
async def _advance_reboot(app, run, connection, transport, connect) -> None:
|
||||
"""Reboot once, and wait for the new version to be what's running. The version
|
||||
is judged against the old one, so a machine that fell back to it fails at the
|
||||
deadline, as does one that never comes back."""
|
||||
token = await client.access_token(connection, transport)
|
||||
url, verify = connection.url, connection.verify_tls
|
||||
if run.reboot_sent_at is None:
|
||||
await _mark_reboot_sent(app, run.id, datetime.now(timezone.utc))
|
||||
try:
|
||||
await client.call_action(url, "hassio", "host_reboot", {}, token, verify, transport)
|
||||
except client.Unreachable:
|
||||
await _mark_reboot_sent(app, run.id, None)
|
||||
await _past_deadline(app, run)
|
||||
return
|
||||
except client.Rejected as exc:
|
||||
await _finish(app, run.id, "failed", f"{run.to_version} was installed, but Home Assistant "
|
||||
f"wouldn't reboot: {exc} It needs a reboot to take effect.")
|
||||
return
|
||||
found = await client.read_os(url, token, verify, connect)
|
||||
if found and found.get("version") not in (None, run.from_version) and not found.get("version_pending"):
|
||||
await _finish(app, run.id, "done", None)
|
||||
return
|
||||
await _past_deadline(app, run)
|
||||
@@ -29,6 +29,10 @@ def setup(app: "Quart") -> None:
|
||||
from steward.core.alerts import names_behind
|
||||
from .routes import name_hottest_sensor
|
||||
names_behind("host_agent", "temp_c_max", name_hottest_sensor)
|
||||
# get_host over MCP marks the sensors the host's temperature leaves out.
|
||||
from steward.mcp.tools.hosts import judge_sensors
|
||||
from .routes import split_temps
|
||||
judge_sensors(lambda host, temps: {k: why for k, (_c, why) in split_temps(host, temps)[1].items()})
|
||||
# A reboot the agent announced is silence with a reason, for a while (#5756).
|
||||
from datetime import datetime, timezone
|
||||
from steward.core.alerts import agents_away
|
||||
|
||||
@@ -22,7 +22,7 @@ from collections import deque
|
||||
from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
|
||||
from datetime import datetime, timezone
|
||||
|
||||
AGENT_VERSION = "1.26.0"
|
||||
AGENT_VERSION = "1.27.0"
|
||||
|
||||
# Default path to the local Docker Engine socket. Overridable via the
|
||||
# `docker_socket` config key; collection is silently skipped if it's absent or
|
||||
@@ -480,7 +480,9 @@ def nfs_delays(cur: dict, prev: dict, dt: float) -> list:
|
||||
|
||||
|
||||
def collect_temps() -> list:
|
||||
"""Return [{label, celsius}] from /sys/class/hwmon (best-effort, may be empty)."""
|
||||
"""Return [{label, celsius, chip}] from /sys/class/hwmon (best-effort, may
|
||||
be empty). `chip` is the hwmon driver's name (coretemp, nvme, nct6798…),
|
||||
which says what a bare label like "Sensor 2" is measuring."""
|
||||
out: list = []
|
||||
try:
|
||||
chips = os.listdir(HWMON_DIR)
|
||||
@@ -511,7 +513,7 @@ def collect_temps() -> list:
|
||||
except OSError:
|
||||
pass
|
||||
name = label or (f"{chip}_{idx}" if chip else idx)
|
||||
out.append({"label": name, "celsius": round(milli / 1000.0, 1)})
|
||||
out.append({"label": name, "celsius": round(milli / 1000.0, 1), "chip": chip})
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -21,11 +21,12 @@ the host it runs on, picks up where it was.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, time, timedelta, timezone
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from steward.core.weekly import WEEKDAYS, describe, fired_slot, is_due, last_slot, next_slot, parse_at # noqa: F401
|
||||
|
||||
from . import maintenance, rebalance, timeline
|
||||
|
||||
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
|
||||
# How long a host may take to be back in service after its job finished.
|
||||
SETTLE_LIMIT = timedelta(minutes=30)
|
||||
# An agent heard from this recently is posting.
|
||||
@@ -33,58 +34,6 @@ POSTING_WITHIN = timedelta(minutes=3)
|
||||
RUN_STATES = ("running", "done", "failed", "stopped")
|
||||
|
||||
|
||||
def _zone(name: str):
|
||||
from steward.core.formatting import zone
|
||||
return zone(name)
|
||||
|
||||
|
||||
def parse_at(at_time: str) -> time:
|
||||
h, m = (int(x) for x in at_time.split(":", 1))
|
||||
if not (0 <= h < 24 and 0 <= m < 60):
|
||||
raise ValueError(at_time)
|
||||
return time(h, m)
|
||||
|
||||
|
||||
def last_slot(plan, now: datetime) -> datetime:
|
||||
"""The most recent scheduled time at or before `now`, in UTC."""
|
||||
tz = _zone(plan.time_zone)
|
||||
local = now.astimezone(tz)
|
||||
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
|
||||
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
|
||||
if slot > local:
|
||||
slot -= timedelta(days=7)
|
||||
return slot.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def fired_slot(plan) -> datetime | None:
|
||||
"""The last slot that fired, or None while none has. A new plan's
|
||||
last_slot_at holds the moment it was made, so its first slot is the
|
||||
next one rather than one already past; that moment is no slot (#5863)."""
|
||||
at = plan.last_slot_at
|
||||
if at is None:
|
||||
return None
|
||||
return at if last_slot(plan, at) == at else None
|
||||
|
||||
|
||||
def next_slot(plan, now: datetime) -> datetime:
|
||||
return last_slot(plan, now) + timedelta(days=7)
|
||||
|
||||
|
||||
def is_due(plan, now: datetime) -> datetime | None:
|
||||
"""The slot to fire now, or None. A slot fires once (last_slot_at), and
|
||||
only while its window is still open: a Steward that was down over the slot
|
||||
starts late, not a day late."""
|
||||
if not plan.enabled:
|
||||
return None
|
||||
slot = last_slot(plan, now)
|
||||
since = plan.last_slot_at or plan.created_at
|
||||
if since is not None and slot <= since:
|
||||
return None
|
||||
if now - slot >= timedelta(minutes=plan.window_minutes):
|
||||
return None
|
||||
return slot
|
||||
|
||||
|
||||
def back_in_service(*, last_seen_at, node, services, job_finished_at, now: datetime,
|
||||
expect_active: bool = True) -> str | None:
|
||||
"""None when the host is back in service, else why not yet. `node` is the
|
||||
@@ -134,13 +83,6 @@ def next_move(run, plan, job, back: str | None, now: datetime) -> tuple[str, str
|
||||
return "settling", back
|
||||
|
||||
|
||||
def describe(plan) -> str:
|
||||
""""Sundays at 03:00 (Europe/London), up to 4 h" for the page."""
|
||||
hours = plan.window_minutes / 60
|
||||
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
|
||||
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
|
||||
|
||||
|
||||
# ── applied to the database: the minute tick, Run now, the page ──────────────
|
||||
|
||||
async def group_hosts(session, group_id: str) -> list:
|
||||
|
||||
@@ -101,10 +101,10 @@ def valid(req) -> bool:
|
||||
|
||||
|
||||
def run(cmd: list, deadline: int = COMMAND_DEADLINE_SECONDS, env: dict | None = None,
|
||||
ok_codes: tuple = (0,), keep: int = OUTPUT_KEPT_CHARS):
|
||||
ok_codes: tuple = (0,), keep: int | None = OUTPUT_KEPT_CHARS):
|
||||
"""(ok, output tail) of a command, bounded by `deadline`. `ok_codes` are
|
||||
the exit codes that mean it worked (zypper says more than 0); `keep` is
|
||||
how much of the output's end is kept."""
|
||||
how much of the output's end is kept, None for all of it."""
|
||||
try:
|
||||
done = subprocess.run(cmd, capture_output=True, text=True, timeout=deadline,
|
||||
env=dict(os.environ, LANG="C", LC_ALL="C", **(env or {})))
|
||||
@@ -113,7 +113,7 @@ def run(cmd: list, deadline: int = COMMAND_DEADLINE_SECONDS, env: dict | None =
|
||||
except OSError as exc:
|
||||
return False, str(exc)
|
||||
out = (done.stdout or "") + (done.stderr or "")
|
||||
return done.returncode in ok_codes, out[-keep:]
|
||||
return done.returncode in ok_codes, out if keep is None else out[-keep:]
|
||||
|
||||
|
||||
REBOOT_MARKER = "/var/run/reboot-required" # Debian and Ubuntu
|
||||
@@ -226,9 +226,11 @@ def do_update() -> dict:
|
||||
"(apt, dnf, yum, zypper, pacman) on this host"}
|
||||
out = ""
|
||||
for label, cmd, deadline, env, ok_codes in update_steps(manager):
|
||||
ok, out = run(cmd, deadline=deadline, env=env, ok_codes=ok_codes)
|
||||
# The whole output: every manager prints its count before the long
|
||||
# log of what it then does, so a tail loses it on a large update.
|
||||
ok, out = run(cmd, deadline=deadline, env=env, ok_codes=ok_codes, keep=None)
|
||||
if not ok:
|
||||
return {"ok": False, "detail": f"{label} failed: " + out}
|
||||
return {"ok": False, "detail": f"{label} failed: " + out[-OUTPUT_KEPT_CHARS:]}
|
||||
changed = upgraded_count(out)
|
||||
if manager == "apt":
|
||||
run(["apt-get", "-y", "autoremove"], env={"DEBIAN_FRONTEND": "noninteractive"})
|
||||
|
||||
@@ -124,13 +124,15 @@ def is_thermometer(host_name: str, label: str, celsius: float) -> bool:
|
||||
|
||||
def hottest_sensor(host_name: str, temps: dict[str, float]) -> list[str]:
|
||||
"""Which sensor sets a host's temperature, as an alert on it says:
|
||||
["Package id 0 at 86°C"], or [] with no thermometer among `temps`."""
|
||||
["CPU · Package id 0 at 86°C"], or [] with no thermometer among `temps`.
|
||||
The chip leads the label when the host's agent has said which it is."""
|
||||
from steward.core.formatting import metric_value
|
||||
real = {label: c for label, c in temps.items() if is_thermometer(host_name, label, c)}
|
||||
if not real:
|
||||
return []
|
||||
from .sensors import sensor_name
|
||||
label = max(real, key=real.get)
|
||||
return [f"{label} at {metric_value('temp_c', real[label])}"]
|
||||
return [f"{sensor_name(host_name, label)} at {metric_value('temp_c', real[label])}"]
|
||||
|
||||
|
||||
async def name_hottest_sensor(session, resource: str) -> list[str]:
|
||||
@@ -374,6 +376,8 @@ def _expand_sample_to_metrics(
|
||||
# counts only readings that can be real (#5759), from sensors that behave
|
||||
# like thermometers (#5783).
|
||||
max_temp: float | None = None
|
||||
from .sensors import note_chips
|
||||
note_chips(host_name, sample.get("temps"))
|
||||
for t in sample.get("temps") or []:
|
||||
c = t.get("celsius")
|
||||
if c is None:
|
||||
@@ -1003,7 +1007,7 @@ def _new_token_pair() -> tuple[str, str]:
|
||||
@host_agent_bp.get("/vitals/<host_id>")
|
||||
@require_role(UserRole.viewer)
|
||||
async def host_vitals(host_id: str):
|
||||
"""Compact live vitals strip (CPU/MEM/DISK/LOAD + pressure) for the Hosts hub.
|
||||
"""Compact live vitals strip (CPU/MEM/DISK/LOAD/temperature + pressure) for the Hosts hub.
|
||||
|
||||
Self-contained fragment, polled by hosts/detail.html. Renders nothing until
|
||||
the agent reports — the Agent panel below then carries provisioning.
|
||||
@@ -1018,15 +1022,16 @@ async def host_vitals(host_id: str):
|
||||
latest = await _latest_metrics_for_host(db, host.name) if reg else {}
|
||||
norms = await host_svc.host_norms(db, host.name) if reg else {}
|
||||
|
||||
# The last 6 h for the at-a-glance sparklines (cpu/mem/load host-level,
|
||||
# The last 6 h for the at-a-glance sparklines (cpu/mem/load/temp host-level,
|
||||
# disk from the root mount), averaged into ~120 points in SQL, so the
|
||||
# line spans the whole window and only what is drawn is fetched.
|
||||
series_raw: dict[str, list[float]] = {"cpu": [], "mem": [], "disk": [], "load": []}
|
||||
series_raw: dict[str, list[float]] = {"cpu": [], "mem": [], "disk": [], "load": [], "temp": []}
|
||||
if reg:
|
||||
since = datetime.now(timezone.utc) - timedelta(hours=6)
|
||||
for key, resource, metric in (("cpu", host.name, "cpu_pct"),
|
||||
("mem", host.name, "mem_used_pct"),
|
||||
("load", host.name, "load_1m"),
|
||||
("temp", host.name, "temp_c_max"),
|
||||
("disk", host.name + ":/", "disk_used_pct")):
|
||||
series_raw[key] = [v for _t, v in await metric_points(
|
||||
db, SOURCE_MODULE, resource, metric, since, target=120)]
|
||||
@@ -1053,9 +1058,16 @@ async def host_vitals(host_id: str):
|
||||
reporting = bool(reg) and ls is not None
|
||||
stale = reporting and (datetime.now(timezone.utc) - ls).total_seconds() > _stale_after_seconds()
|
||||
cpu, mem = hostlvl.get("cpu_pct"), hostlvl.get("mem_used_pct")
|
||||
# The hottest sensor that counts (temp_c_max leaves out unwired inputs),
|
||||
# named so the tile says what it is reading.
|
||||
temp = hostlvl.get("temp_c_max")
|
||||
prefix = host.name + ":temp:"
|
||||
hottest = hottest_sensor(host.name, {r[len(prefix):]: m["temp_c"] for r, m in latest.items()
|
||||
if r.startswith(prefix) and m.get("temp_c") is not None})
|
||||
thresholds = current_app.config.get("THRESHOLDS", {})
|
||||
worst = worst_reading((("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", disk_root, "disk"),
|
||||
("Inodes on /", inodes_root, "disk"), ("Load per core", load_per_core, "load")),
|
||||
("Inodes on /", inodes_root, "disk"), ("Load per core", load_per_core, "load"),
|
||||
("Temperature", temp, "temp")),
|
||||
thresholds, host_svc.host_normals(norms, thresholds))
|
||||
# Each tile's grade; CPU, memory and load within the host's own normal
|
||||
# for the hour read healthy, and say what is usual (#5742).
|
||||
@@ -1068,7 +1080,7 @@ async def host_vitals(host_id: str):
|
||||
return await render_template(
|
||||
"_host_vitals.html",
|
||||
reg=reg, cpu=cpu, mem=mem, disk_root=disk_root, load_per_core=load_per_core,
|
||||
inodes_root=inodes_root,
|
||||
inodes_root=inodes_root, temp=temp, hottest=hottest[0].rsplit(" at ", 1)[0] if hottest else None,
|
||||
sparks=sparks, psi=psi, reporting=reporting, stale=stale, worst=worst,
|
||||
grades=grades, usual=usual,
|
||||
clock=clock_words(reg.clock_offset_s if reg else None, current_app.config.get("THRESHOLDS", {})),
|
||||
|
||||
@@ -53,6 +53,51 @@ def unwired(host_name: str) -> dict[str, str]:
|
||||
return _unwired.get(host_name, {})
|
||||
|
||||
|
||||
# {host name: {sensor label: hwmon chip}}, as the host's latest sample said.
|
||||
# A sensor's metric is keyed by its label alone; the chip only names it.
|
||||
_chips: dict[str, dict[str, str]] = {}
|
||||
|
||||
# What each hwmon driver measures, in the words a page uses. A driver not
|
||||
# listed is shown by its own name.
|
||||
_CHIP_WORDS = (
|
||||
(re.compile(r"^(coretemp|k10temp|zenpower|cpu_thermal)$"), "CPU"),
|
||||
(re.compile(r"^nvme$"), "NVMe"),
|
||||
(re.compile(r"^drivetemp$"), "Drive"),
|
||||
(re.compile(r"^(nct\d+|it\d+|w83\w+|f71\w+)$"), "Board"),
|
||||
(re.compile(r"^pch_\w+$"), "Chipset"),
|
||||
(re.compile(r"^(amdgpu|radeon|nouveau|i915|xe)$"), "GPU"),
|
||||
(re.compile(r"^(spd5118|jc42)$"), "Memory"),
|
||||
(re.compile(r"^iwlwifi"), "Wi-Fi"),
|
||||
(re.compile(r"^acpitz$"), "ACPI"),
|
||||
)
|
||||
|
||||
|
||||
def note_chips(host_name: str, temps) -> None:
|
||||
"""Remember which chip each of a host's sensors is on, from a sample's
|
||||
temps. An agent that doesn't say leaves what is known as it is."""
|
||||
seen = {str(t["label"]): str(t["chip"]) for t in temps or [] if t.get("chip")}
|
||||
if seen:
|
||||
_chips[host_name] = seen
|
||||
|
||||
|
||||
def chip_word(chip: str) -> str:
|
||||
"""What a hwmon chip measures ("nvme" → "NVMe"), or its own name."""
|
||||
for pattern, word in _CHIP_WORDS:
|
||||
if pattern.match(chip):
|
||||
return word
|
||||
return chip
|
||||
|
||||
|
||||
def sensor_name(host_name: str, label: str) -> str:
|
||||
"""A sensor as a page names it: its chip, then its label, as in
|
||||
"NVMe · Sensor 2". The label alone when the chip isn't known or the
|
||||
label already carries it ("nvme_temp1")."""
|
||||
chip = _chips.get(host_name, {}).get(label)
|
||||
if not chip or label.lower().startswith(chip.lower()):
|
||||
return label
|
||||
return f"{chip_word(chip)} · {label}"
|
||||
|
||||
|
||||
def reference_label(labels) -> str | None:
|
||||
"""The sensor that reads the host's CPU, or None when it has none."""
|
||||
labels = sorted(labels)
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
{# Host health header — live polled into the host hub: live or stale, the worst
|
||||
reading against Settings → Thresholds, last seen, then CPU / memory / disk /
|
||||
load tiles graded by the same cutoffs (`grades`; CPU, memory and load within
|
||||
load tiles, and the hottest sensor that counts with its name, graded by the same cutoffs (`grades`; CPU, memory and load within
|
||||
the host's own normal for the hour read healthy, and `usual` says so). Renders nothing until the agent
|
||||
reports (the Agent panel below then shows provisioning). #}
|
||||
{% from "_macros.html" import status_badge, stat_tile %}
|
||||
{% macro pct(v) %}{{ '%.0f%%'|format(v) if v is not none else '—' }}{% endmacro %}
|
||||
{% macro deg(v) %}{{ '%.0f°C'|format(v) if v is not none else '—' }}{% endmacro %}
|
||||
{% set grades = grades or {} %}
|
||||
{% if reporting %}
|
||||
<section class="card host-health" aria-label="Host health">
|
||||
@@ -12,7 +13,7 @@
|
||||
{{ status_badge('stale' if stale else 'up', 'Stale' if stale else 'Live') }}
|
||||
<span class="host-health-summary">
|
||||
{%- if stale %}No report since {{ reg.last_seen_at|ago }}; the readings below are from then.
|
||||
{%- elif worst %}<span class="tone-{{ worst[2] }}">{{ worst[0] }} {{ pct(worst[1]) }}</span> is past its {{ "critical" if worst[2] == "crit" else "warning" }} threshold.
|
||||
{%- elif worst %}<span class="tone-{{ worst[2] }}">{{ worst[0] }} {{ deg(worst[1]) if worst[0] == "Temperature" else pct(worst[1]) }}</span> is past its {{ "critical" if worst[2] == "crit" else "warning" }} threshold.
|
||||
{%- elif usual %}Every reading is within its thresholds or its usual level for this hour.
|
||||
{%- else %}Every reading is within its thresholds.{% endif -%}
|
||||
</span>
|
||||
@@ -21,14 +22,20 @@
|
||||
{%- if reg.agent_version %}{{ " · " if not stale }}agent v{{ reg.agent_version }}{% endif -%}
|
||||
{%- if clock and clock[0] %} · <span{% if clock[1] in ('warn', 'crit') %} class="tone-{{ clock[1] }}" title="Further from Steward's clock than Settings → Thresholds allows: check what keeps this host's time in sync"{% endif %}>{{ clock[0] }}</span>{% endif -%}
|
||||
</span>
|
||||
{# Charts, every mount, interface, disk and temperature sensor. #}
|
||||
<a href="/plugins/host_agent/{{ reg.host_id }}/" class="btn btn-sm btn-ghost" style="margin-left:auto;">Full metrics →</a>
|
||||
</div>
|
||||
<div class="stat-strip host-health-tiles" style="margin-bottom:0;">
|
||||
<div class="stat-strip host-health-tiles" style="margin-bottom:0;--tiles:{{ 5 if temp is not none else 4 }};">
|
||||
{{ stat_tile("CPU", pct(cpu), state=(grades.cpu or [status_class(cpu, 'cpu')])[0], hint=sparks.cpu|safe) }}
|
||||
{{ stat_tile("Memory", pct(mem), state=(grades.mem or [status_class(mem, 'mem')])[0], hint=sparks.mem|safe) }}
|
||||
{%- set inodes_worse = inodes_root is defined and inodes_root is not none and (disk_root is none or inodes_root > disk_root) %}
|
||||
{{ stat_tile("Disk /", pct(disk_root) ~ (" · inodes " ~ pct(inodes_root) if inodes_worse else ""),
|
||||
state=(grades.disk or [status_class(disk_root, 'disk')])[0], hint=sparks.disk|safe) }}
|
||||
{{ stat_tile("Load per core", pct(load_per_core), state=(grades.load or [status_class(load_per_core, 'load')])[0], hint=sparks.load|safe) }}
|
||||
{% if temp is not none %}
|
||||
{%- set temp_hint %}{{ sparks.temp|safe }}{% if hottest %}<span class="stat-tile-sensor" title="{{ hottest }}">{{ hottest }}</span>{% endif %}{% endset %}
|
||||
{{ stat_tile("Temperature", deg(temp), state=status_class(temp, 'temp'), hint=temp_hint) }}
|
||||
{% endif %}
|
||||
</div>
|
||||
{% if usual and not stale %}
|
||||
<p class="field-hint" style="margin-top:0.6rem;">{{ usual | join("; ") | capitalize }}.</p>
|
||||
|
||||
@@ -14,10 +14,9 @@
|
||||
</div>
|
||||
|
||||
{% if reporting %}
|
||||
{# Reporting: live vitals are shown by the vitals strip above; this panel is
|
||||
the way to the full metrics, with the lifecycle actions folded under
|
||||
"Manage agent" (#5278). #}
|
||||
<a href="/plugins/host_agent/{{ host.id }}/" class="btn btn-sm btn-ghost">Full metrics →</a>
|
||||
{# Reporting: live vitals, and the way to the full metrics, are in the
|
||||
vitals strip at the top of the page; this panel holds what the host has
|
||||
waiting, with the lifecycle actions folded under "Manage agent" (#5278). #}
|
||||
|
||||
{# What the host has waiting, from the agent's hourly check (#5765): a
|
||||
needed reboot first, then the updates. #}
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""A target whose token was refused stops being scraped until it is saved
|
||||
|
||||
Revision ID: prometheus_003_refused
|
||||
Revises: prometheus_002_templates
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "prometheus_003_refused"
|
||||
down_revision: Union[str, None] = "prometheus_002_templates"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("prometheus_targets", sa.Column("refused", sa.Boolean, nullable=False, server_default=sa.false()))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("prometheus_targets", "refused")
|
||||
@@ -0,0 +1,23 @@
|
||||
"""A target can take its token from another plugin's sign-in
|
||||
|
||||
Revision ID: prometheus_004_token_from
|
||||
Revises: prometheus_003_refused
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "prometheus_004_token_from"
|
||||
down_revision: Union[str, None] = "prometheus_003_refused"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("prometheus_targets", "token_from")
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Every target holds its own token: one that took it from a sign-in goes
|
||||
|
||||
Revision ID: prometheus_005_own_tokens
|
||||
Revises: prometheus_004_token_from
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "prometheus_005_own_tokens"
|
||||
down_revision: Union[str, None] = "prometheus_004_token_from"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Left in place, such a target would ask with no token, and some services ban
|
||||
# an address for that. Its newest values go with it; history ages out. On a new
|
||||
# database the metric tables may not exist yet, and there is nothing to remove.
|
||||
if sa.inspect(op.get_bind()).has_table("plugin_metric_latest"):
|
||||
op.execute("""
|
||||
DELETE FROM plugin_metric_latest l USING prometheus_targets t
|
||||
WHERE t.token_from IS NOT NULL AND l.source_module = 'prometheus'
|
||||
AND (l.resource_name = t.name OR left(l.resource_name, length(t.name) + 1) = t.name || ':')""")
|
||||
op.execute("DELETE FROM prometheus_targets WHERE token_from IS NOT NULL")
|
||||
op.drop_column("prometheus_targets", "token_from")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True))
|
||||
@@ -17,7 +17,8 @@ class PrometheusTarget(Base):
|
||||
patterns ("fabledarchives_*") an admin chooses: Steward does the
|
||||
filtering, the endpoint serves everything. The last_* columns are the
|
||||
newest scrape's outcome, written by the scraper, so a page can say why a
|
||||
target isn't answering without asking it.
|
||||
target isn't answering without asking it. `refused` says its token was
|
||||
refused, which stops the scraping until the target is saved or resumed.
|
||||
"""
|
||||
__tablename__ = "prometheus_targets"
|
||||
|
||||
@@ -31,6 +32,7 @@ class PrometheusTarget(Base):
|
||||
drop: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
series_cap: Mapped[int] = mapped_column(Integer, nullable=False, default=1000)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
refused: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
|
||||
last_scraped_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
@@ -153,6 +153,7 @@ async def save_target(target_id: str):
|
||||
# Where it is or what is kept may have changed: the last outcome no
|
||||
# longer describes it, so the page says "not scraped yet" until the next scrape.
|
||||
target.last_scraped_at = target.last_ok_at = target.last_error = None
|
||||
target.refused = False
|
||||
target.last_kept = target.last_dropped = target.last_capped = 0
|
||||
if old_name != target.name:
|
||||
# Its history is stored under its name, and its labelled series
|
||||
@@ -188,6 +189,7 @@ async def toggle_target(target_id: str):
|
||||
# Resumed: it reads "not scraped yet" until the next scrape,
|
||||
# rather than stale from before the pause.
|
||||
target.last_scraped_at = target.last_error = None
|
||||
target.refused = False
|
||||
enabled, name = target.enabled, target.name
|
||||
# A rate needs two readings in a row; one from before the pause isn't one.
|
||||
from . import scheduler
|
||||
|
||||
@@ -40,7 +40,7 @@ def make_scrape_task(app: "Quart") -> ScheduledTask:
|
||||
def is_due(target, now: datetime) -> bool:
|
||||
"""Whether a target's interval has elapsed since its last scrape, with
|
||||
half a tick of slack so tick jitter doesn't push it back a whole tick."""
|
||||
if not target.enabled:
|
||||
if not target.enabled or target.refused:
|
||||
return False
|
||||
if target.last_scraped_at is None:
|
||||
return True
|
||||
@@ -115,7 +115,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None:
|
||||
from steward.core.alerts import MetricBatch
|
||||
|
||||
from .models import PrometheusTarget
|
||||
from .scraper import ScrapeError
|
||||
from .scraper import ScrapeError, TokenRefused
|
||||
from .service import (IS_UP, OWN_METRICS, SCRAPE_MS, SERIES_CAPPED, SERIES_KEPT, SOURCE,
|
||||
map_families)
|
||||
|
||||
@@ -132,6 +132,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None:
|
||||
if not isinstance(result, ScrapeError):
|
||||
logger.warning("Prometheus scrape of %s failed: %r", target.name, result)
|
||||
row.last_error = str(result) or "The scrape failed."
|
||||
row.refused = isinstance(result, TokenRefused)
|
||||
batch.add(source_module=SOURCE, resource_name=row.name, metric_name=IS_UP,
|
||||
value=0.0, recorded_at=now)
|
||||
continue
|
||||
|
||||
@@ -18,6 +18,11 @@ class ScrapeError(Exception):
|
||||
"""Why a scrape got nothing, said to the person."""
|
||||
|
||||
|
||||
class TokenRefused(ScrapeError):
|
||||
"""The endpoint refused the token. Asking again won't change that, and
|
||||
Home Assistant, for one, bans an address after a few refusals."""
|
||||
|
||||
|
||||
async def scrape(url: str, token: str = "", verify_tls: bool = True,
|
||||
transport: httpx.AsyncBaseTransport | None = None) -> dict[str, prometheus.Family]:
|
||||
"""The endpoint's families. `transport` stands in for the network in tests."""
|
||||
@@ -29,8 +34,8 @@ async def scrape(url: str, token: str = "", verify_tls: bool = True,
|
||||
transport=transport, follow_redirects=False) as client:
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code in (401, 403):
|
||||
raise ScrapeError(f"Answered {response.status_code}: the token was refused. "
|
||||
"Check the token, or make a new one on the service.")
|
||||
raise TokenRefused(f"Answered {response.status_code}: the token was refused. "
|
||||
"Check the token, or make a new one on the service.")
|
||||
if response.status_code != 200:
|
||||
raise ScrapeError(f"Answered {response.status_code} {response.reason_phrase}, not metrics.")
|
||||
body = bytearray()
|
||||
|
||||
@@ -271,6 +271,9 @@ class TargetState:
|
||||
def target_state(target: PrometheusTarget, now: datetime) -> TargetState:
|
||||
if not target.enabled:
|
||||
return TargetState("paused", "Paused: not scraped until it's resumed.")
|
||||
if target.refused:
|
||||
return TargetState("down", f"{target.last_error} Steward stopped asking, as some services ban an "
|
||||
"address after a few refusals. Save a new token to start again.")
|
||||
if target.last_scraped_at is None:
|
||||
return TargetState("pending", "Not scraped yet.")
|
||||
if (now - target.last_scraped_at).total_seconds() > STALE_SCRAPES * max(target.interval_seconds, 1):
|
||||
|
||||
@@ -410,7 +410,7 @@ async def rows():
|
||||
phases.mark("window")
|
||||
# The routers' metric anomalies, last 30 days (#5741).
|
||||
anomalies = await anom.views(db, await anom.list_anomalies(
|
||||
db, source="traefik", since=now - timedelta(days=30), limit=20), links=False)
|
||||
db, source="traefik", since=now - timedelta(days=30), limit=100), links=False)
|
||||
for r in current:
|
||||
points = history[r["name"]]
|
||||
r["since"], r["since_before"] = (problem_since(points, thresholds) if r["state"] != "ok"
|
||||
|
||||
@@ -17,6 +17,8 @@ dependencies = [
|
||||
"packaging>=24.0",
|
||||
"click>=8.0",
|
||||
"httpx>=0.27",
|
||||
# Floor: websockets.asyncio.client first ships in 13.
|
||||
"websockets>=13",
|
||||
"cryptography>=42",
|
||||
# Floor, not a cap: MCPServer (mcp.server.mcpserver) and the mcp.types
|
||||
# mirror that steward/mcp/ is written against first ship in SDK 2.2.
|
||||
|
||||
@@ -284,15 +284,24 @@ async def alert_history():
|
||||
@require_role(UserRole.viewer)
|
||||
async def anomalies():
|
||||
"""Metric anomalies (#5740): those to review, newest occurrence first,
|
||||
then the reviewed and lapsed ones under day dividers."""
|
||||
then the reviewed and lapsed ones under day dividers. `source` and
|
||||
`resource` narrow it to one resource's, its parts included: a resource's
|
||||
page links here with them."""
|
||||
from urllib.parse import urlencode
|
||||
from steward.audit.service import mark_day_groups
|
||||
from steward.core import anomalies as anom
|
||||
from steward.core import formatting as fmt
|
||||
source = request.args.get("source", "").strip() or None
|
||||
resource = request.args.get("resource", "").strip() or None
|
||||
only = {"source": source, "resource": resource}
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
to_review = await anom.views(db, await anom.list_anomalies(db, (anom.TO_REVIEW,)))
|
||||
to_review = await anom.views(db, await anom.list_anomalies(db, (anom.TO_REVIEW,), **only))
|
||||
past = await anom.views(db, await anom.list_anomalies(
|
||||
db, (anom.NORMAL, anom.REAL, anom.LAPSED), limit=PAGE_SIZE * 4))
|
||||
db, (anom.NORMAL, anom.REAL, anom.LAPSED), limit=PAGE_SIZE * 4, **only))
|
||||
days = int(await get_setting(db, "anomalies.review_days") or 14)
|
||||
here = "/alerts/anomalies"
|
||||
if source or resource:
|
||||
here += "?" + urlencode({k: v for k, v in only.items() if v})
|
||||
tz = fmt.zone(session.get("tz"))
|
||||
to_review = [{"v": v} for v in to_review]
|
||||
past = [{"v": v} for v in past]
|
||||
@@ -300,7 +309,7 @@ async def anomalies():
|
||||
mark_day_groups(to_review, tz, at=at)
|
||||
mark_day_groups(past, tz, at=at)
|
||||
return await render_template("alerts/anomalies.html", to_review=to_review, past=past,
|
||||
review_days=days)
|
||||
review_days=days, source=source, resource=resource, here=here)
|
||||
|
||||
|
||||
@alerts_bp.post("/anomalies/<anomaly_id>/review")
|
||||
|
||||
@@ -214,6 +214,10 @@ ACTIONS = {
|
||||
"prometheus.template_created": "added Prometheus widget template",
|
||||
"prometheus.template_updated": "changed Prometheus widget template",
|
||||
"prometheus.template_deleted": "deleted Prometheus widget template",
|
||||
"homeassistant.connection_saved": "signed Steward in to Home Assistant",
|
||||
"homeassistant.connection_removed": "removed the Home Assistant connection",
|
||||
"homeassistant.update_started": "started a Home Assistant update",
|
||||
"homeassistant.schedule_saved": "set Home Assistant's update schedule",
|
||||
"syslog.source_added": "added syslog source",
|
||||
"syslog.source_updated": "changed syslog source",
|
||||
"syslog.source_removed": "removed syslog source",
|
||||
@@ -244,7 +248,7 @@ AREAS = (("auth.", "Sign-ins"), ("account.", "Accounts"), ("user.", "Users"), ("
|
||||
("ansible.run", "Playbook runs"), ("ansible.", "Ansible (all)"),
|
||||
("settings.", "Settings"), ("plugin.", "Plugins"), ("api_token.", "AI access"),
|
||||
("docker.", "Docker"), ("snmp.", "SNMP"), ("prometheus.", "Prometheus"),
|
||||
("syslog.", "Syslog"), ("backup.", "Backups"), ("timeline.", "Timeline"),
|
||||
("syslog.", "Syslog"), ("homeassistant.", "Home Assistant"), ("backup.", "Backups"), ("timeline.", "Timeline"),
|
||||
("ticket.", "Tickets"))
|
||||
|
||||
|
||||
|
||||
@@ -29,6 +29,8 @@ KEPT: frozenset[str] = frozenset({
|
||||
# Syslog senders: which were added, under what name, and which ignored.
|
||||
"syslog_sources",
|
||||
"syslog_patterns",
|
||||
"homeassistant_connection",
|
||||
"homeassistant_schedule",
|
||||
})
|
||||
|
||||
# Rows that are measurements, logs, history or in-flight work: left out.
|
||||
@@ -67,6 +69,7 @@ EXCLUDED: frozenset[str] = frozenset({
|
||||
"unifi_events", "unifi_fw_rules", "unifi_networks", "unifi_port_forwards",
|
||||
"unifi_speedtest_results", "unifi_wan_stats", "unifi_wlan_stats",
|
||||
"syslog_lines",
|
||||
"homeassistant_updates",
|
||||
})
|
||||
|
||||
# ── What a check compares (#5823) ────────────────────────────────────────────
|
||||
@@ -89,8 +92,11 @@ ACTIVITY: dict[str, frozenset[str]] = {
|
||||
"monitors": frozenset({"last_checked_at"}),
|
||||
"ansible_schedules": frozenset({"last_run_at", "last_run_id", "last_error"}),
|
||||
"snmp_devices": frozenset({"last_polled_at", "last_ok_at", "last_error", "missing"}),
|
||||
"prometheus_targets": frozenset({"last_scraped_at", "last_ok_at", "last_error", "last_kept",
|
||||
"prometheus_targets": frozenset({"refused", "last_scraped_at", "last_ok_at", "last_error", "last_kept",
|
||||
"last_dropped", "last_capped"}),
|
||||
"homeassistant_connection": frozenset({"refused", "last_polled_at", "last_ok_at", "last_error",
|
||||
"updates", "backup", "health"}),
|
||||
"homeassistant_schedule": frozenset({"last_slot_at", "stopped"}),
|
||||
# What the sender last sent; its address, name, state and host are setup.
|
||||
"syslog_sources": frozenset({"hostname_seen", "first_seen_at", "last_seen_at", "lines_total",
|
||||
"last_severity", "last_line"}),
|
||||
@@ -121,5 +127,6 @@ AREAS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("Ansible", ("ansible_targets", "ansible_groups", "ansible_target_groups", "ansible_schedules")),
|
||||
("Dashboards", ("dashboards", "dashboard_widgets", "dashboard_share_tokens")),
|
||||
("Plugins", ("snmp_devices", "prometheus_targets", "prometheus_widget_templates",
|
||||
"unifi_known_clients", "syslog_sources", "syslog_patterns")),
|
||||
"unifi_known_clients", "syslog_sources", "syslog_patterns",
|
||||
"homeassistant_connection", "homeassistant_schedule")),
|
||||
)
|
||||
|
||||
+57
-19
@@ -7,7 +7,10 @@ departed. That opens an occurrence once it is also outside what the normal
|
||||
records in height or in time: higher than the metric usually goes (Norm.
|
||||
usual_height), or held, with most of the last HELD above the line. A burst of
|
||||
a few minutes within that lifts the average for a while and opens nothing. The
|
||||
occurrence ends when the level comes back under the line.
|
||||
height is judged with the window's highest reading left out, so one slow
|
||||
request or one busy sample can't carry a window past it, and a held level needs
|
||||
at least HELD_FEWEST readings behind its median: a lone reading never opens
|
||||
one. The occurrence ends when the level comes back under the line.
|
||||
|
||||
What the operator ruled (System 182 "Metric Norms and Anomalies"):
|
||||
- A departure is a warning, marked as a metric anomaly, and stays "to review"
|
||||
@@ -123,30 +126,58 @@ async def levels_now(session, now: datetime) -> dict[tuple[str, str, str], float
|
||||
return {(s, r, m): float(level) for s, r, m, level in rows}
|
||||
|
||||
|
||||
# The median, as a burst moves an average of the same stretch but not this.
|
||||
_HELD_LEVELS = text("""
|
||||
SELECT k.s, k.r, k.m, v.held
|
||||
def _per_key(aggregate: str):
|
||||
"""A read of `aggregate` over each given key's readings in [:since, :now].
|
||||
A key whose aggregate is NULL is left out."""
|
||||
return text(f"""
|
||||
SELECT k.s, k.r, k.m, v.x
|
||||
FROM unnest(:sources, :resources, :metrics) AS k(s, r, m)
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) AS held FROM plugin_metrics p
|
||||
SELECT {aggregate} AS x FROM plugin_metrics p
|
||||
WHERE p.source_module = k.s AND p.resource_name = k.r AND p.metric_name = k.m
|
||||
AND p.recorded_at >= :since AND p.recorded_at <= :now
|
||||
) v
|
||||
WHERE v.held IS NOT NULL
|
||||
WHERE v.x IS NOT NULL
|
||||
""").bindparams(bindparam("sources", type_=ARRAY(String)), bindparam("resources", type_=ARRAY(String)),
|
||||
bindparam("metrics", type_=ARRAY(String)))
|
||||
|
||||
|
||||
async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]:
|
||||
"""{(source, resource, metric): the median of its last HELD}. Asked only of
|
||||
keys that have departed and stayed within their usual peak, so the longer
|
||||
read isn't paid for the rest."""
|
||||
# The median, as a burst moves an average of the same stretch but not this.
|
||||
# Fewer than HELD_FEWEST readings in the stretch are not a held level.
|
||||
HELD_FEWEST = 3
|
||||
_HELD_LEVELS = _per_key(
|
||||
f"CASE WHEN count(*) >= {HELD_FEWEST} THEN percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) END")
|
||||
# The average with the single highest reading left out; NULL for one reading.
|
||||
_TOPLESS_LEVELS = _per_key("CASE WHEN count(*) > 1 THEN (sum(p.value) - max(p.value)) / (count(*) - 1) END")
|
||||
|
||||
|
||||
async def _read_per_key(session, query, keys: list[tuple[str, str, str]], since: datetime,
|
||||
now: datetime) -> dict[tuple[str, str, str], float]:
|
||||
if not keys:
|
||||
return {}
|
||||
rows = await session.execute(_HELD_LEVELS, {
|
||||
rows = await session.execute(query, {
|
||||
"sources": [k[0] for k in keys], "resources": [k[1] for k in keys], "metrics": [k[2] for k in keys],
|
||||
"since": now - HELD, "now": now})
|
||||
return {(s, r, m): float(held) for s, r, m, held in rows}
|
||||
"since": since, "now": now})
|
||||
return {(s, r, m): float(x) for s, r, m, x in rows}
|
||||
|
||||
|
||||
async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]:
|
||||
"""{(source, resource, metric): the median of its last HELD}, for keys with
|
||||
at least HELD_FEWEST readings in it. Asked only of keys that have departed
|
||||
and not opened by height, so the longer read isn't paid for the rest."""
|
||||
return await _read_per_key(session, _HELD_LEVELS, keys, now - HELD, now)
|
||||
|
||||
|
||||
async def topless_levels(session, keys: list[tuple[str, str, str]],
|
||||
now: datetime) -> dict[tuple[str, str, str], float]:
|
||||
"""{(source, resource, metric): its average over the last WINDOW without
|
||||
the window's highest reading}. A key with one reading in the window is
|
||||
left out: one reading is never a level."""
|
||||
return await _read_per_key(session, _TOPLESS_LEVELS, keys, now - WINDOW, now)
|
||||
|
||||
|
||||
def _past_height(level: float, norm: Norm, floor: float) -> bool:
|
||||
return level > norm.usual_height and departs(level, norm, floor)
|
||||
|
||||
|
||||
async def detect(session, now: datetime | None = None) -> dict:
|
||||
@@ -166,10 +197,17 @@ async def detect(session, now: datetime | None = None) -> dict:
|
||||
}
|
||||
opened = closed = 0
|
||||
changed: list[MetricAnomaly] = [] # opened, joined or ended: their timeline entries follow
|
||||
out_now = {key for key, level in levels.items()
|
||||
if key in norms and departs(level, norms[key], LEARNED[(key[0], key[2])].least_departure)}
|
||||
held = await held_levels(session, [key for key in out_now
|
||||
if key not in open_now and levels[key] <= norms[key].usual_height], now)
|
||||
def floor(key):
|
||||
return LEARNED[(key[0], key[2])].least_departure
|
||||
|
||||
out_now = {key for key, level in levels.items() if key in norms and departs(level, norms[key], floor(key))}
|
||||
waiting = [key for key in out_now if key not in open_now]
|
||||
# Past the usual height only if it stays past it without its highest
|
||||
# reading: one slow request lifts a window's average on its own.
|
||||
tall = [key for key in waiting if levels[key] > norms[key].usual_height]
|
||||
topless = await topless_levels(session, tall, now)
|
||||
past_height = {key for key in tall if key in topless and _past_height(topless[key], norms[key], floor(key))}
|
||||
held = await held_levels(session, [key for key in waiting if key not in past_height], now)
|
||||
for key, level in levels.items():
|
||||
norm = norms.get(key)
|
||||
out = key in out_now
|
||||
@@ -182,10 +220,10 @@ async def detect(session, now: datetime | None = None) -> dict:
|
||||
_close(anomaly, span, now)
|
||||
changed.append(anomaly)
|
||||
closed += 1
|
||||
elif out and level > norm.usual_height:
|
||||
elif key in past_height:
|
||||
changed.append(await _occur(session, key, level, norm, now, WINDOW))
|
||||
opened += 1
|
||||
elif out and departs(held.get(key, 0.0), norm, LEARNED[(key[0], key[2])].least_departure):
|
||||
elif out and departs(held.get(key, 0.0), norm, floor(key)):
|
||||
changed.append(await _occur(session, key, level, norm, now, HELD))
|
||||
opened += 1
|
||||
# An open occurrence whose key stopped reporting, lost its normal or fell
|
||||
|
||||
@@ -298,6 +298,8 @@ def metric_value(name: str, value: Any) -> str:
|
||||
# its underscores spaced (SNMP's OID labels are the operator's own words).
|
||||
METRICS: dict[str, tuple[str, str]] = {
|
||||
"is_up": ("Up or down", "whether the check passed"),
|
||||
"backup_age_hours": ("Age of last backup (hours)", "hours since Home Assistant's last successful automatic backup"),
|
||||
"updates_waiting": ("Updates waiting", "Core, OS and add-on updates Home Assistant has waiting"),
|
||||
"is_healthy": ("Health check", "the container's own HEALTHCHECK result"),
|
||||
"response_ms": ("Response time", "how long the check took"),
|
||||
"ingest_posts": ("Agent posts per minute", "host-agent reports Steward took in the last minute"),
|
||||
@@ -421,6 +423,7 @@ SOURCES: dict[str, str] = {
|
||||
"http": "HTTP monitors", "steward": "Steward itself", "host_agent": "Host agent",
|
||||
"docker": "Docker containers", "traefik": "Traefik routers", "unifi": "UniFi",
|
||||
"snmp": "SNMP devices", "prometheus": "Prometheus targets", "syslog": "Syslog patterns",
|
||||
"homeassistant": "Home Assistant",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -181,6 +181,7 @@ DEFAULTS: dict[str, Any] = {
|
||||
"plugin.snmp": {"enabled": True},
|
||||
"plugin.prometheus": {"enabled": True},
|
||||
"plugin.syslog": {"enabled": True},
|
||||
"plugin.homeassistant": {"enabled": True},
|
||||
# OIDC single-sign-on
|
||||
"oidc.enabled": False,
|
||||
"oidc.discovery_url": "",
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# steward/core/weekly.py
|
||||
"""A weekly slot: a weekday and a time in a time zone, with a window after it. A
|
||||
slot fires once, and only while its window is open, so a Steward that was down
|
||||
over the slot starts late rather than a day late. A schedule is anything with
|
||||
weekday, at_time, time_zone, window_minutes, enabled, last_slot_at and created_at."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, time, timedelta, timezone
|
||||
|
||||
from steward.core.formatting import zone
|
||||
|
||||
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
|
||||
|
||||
|
||||
def parse_at(at_time: str) -> time:
|
||||
h, m = (int(x) for x in at_time.split(":", 1))
|
||||
if not (0 <= h < 24 and 0 <= m < 60):
|
||||
raise ValueError(at_time)
|
||||
return time(h, m)
|
||||
|
||||
|
||||
def last_slot(plan, now: datetime) -> datetime:
|
||||
"""The most recent scheduled time at or before `now`, in UTC."""
|
||||
tz = zone(plan.time_zone)
|
||||
local = now.astimezone(tz)
|
||||
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
|
||||
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
|
||||
if slot > local:
|
||||
slot -= timedelta(days=7)
|
||||
return slot.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def fired_slot(plan) -> datetime | None:
|
||||
"""The last slot that fired, or None while none has. A new schedule's
|
||||
last_slot_at holds the moment it was made, so its first slot is the
|
||||
next one rather than one already past; that moment is no slot."""
|
||||
at = plan.last_slot_at
|
||||
if at is None:
|
||||
return None
|
||||
return at if last_slot(plan, at) == at else None
|
||||
|
||||
|
||||
def next_slot(plan, now: datetime) -> datetime:
|
||||
return last_slot(plan, now) + timedelta(days=7)
|
||||
|
||||
|
||||
def is_due(plan, now: datetime) -> datetime | None:
|
||||
"""The slot to fire now, or None."""
|
||||
if not plan.enabled:
|
||||
return None
|
||||
slot = last_slot(plan, now)
|
||||
since = plan.last_slot_at or plan.created_at
|
||||
if since is not None and slot <= since:
|
||||
return None
|
||||
if now - slot >= timedelta(minutes=plan.window_minutes):
|
||||
return None
|
||||
return slot
|
||||
|
||||
|
||||
def describe(plan) -> str:
|
||||
""""Sundays at 03:00 (Europe/London), up to 4 h" for a page."""
|
||||
hours = plan.window_minutes / 60
|
||||
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
|
||||
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
|
||||
@@ -132,10 +132,11 @@ async def host_detail(host_id: str):
|
||||
linkable_targets = (await db.execute(
|
||||
select(AnsibleTarget).where(AnsibleTarget.host_id.is_(None))
|
||||
.order_by(AnsibleTarget.name))).scalars().all()
|
||||
# The host's metric anomalies, its disks and interfaces included (#5740).
|
||||
# The host's metric anomalies, its disks and interfaces included
|
||||
# (#5740). The card counts those to review, so the limit is generous.
|
||||
anomalies = await anom.views(db, await anom.list_anomalies(
|
||||
db, source="host_agent", resource=host.name,
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=10), links=False)
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=100), links=False)
|
||||
|
||||
return await render_template(
|
||||
"hosts/detail.html",
|
||||
|
||||
@@ -13,10 +13,10 @@ each argument's empty value means, and what comes back.
|
||||
from __future__ import annotations
|
||||
|
||||
from . import (alerts, anomalies, audit, automation, backups, hosts, metrics, monitors, overview, session,
|
||||
tickets, timeline)
|
||||
thresholds, tickets, timeline)
|
||||
|
||||
_MODULES = (session, overview, hosts, monitors, alerts, anomalies, metrics, audit, timeline, tickets,
|
||||
automation, backups)
|
||||
_MODULES = (session, overview, hosts, monitors, alerts, anomalies, thresholds, metrics, audit, timeline,
|
||||
tickets, automation, backups)
|
||||
|
||||
|
||||
def register_all(register_tool) -> None:
|
||||
|
||||
@@ -293,8 +293,9 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
|
||||
departed there (usual hold ×1.25, and at least `floor` above it). A
|
||||
departure opens an anomaly once it passes both `usual_peak` and
|
||||
`usual_anywhere` (how high the metric has gone in most hours of the day,
|
||||
one figure for the whole key), or when the median of the last hour is over
|
||||
`flag_above` too.
|
||||
one figure for the whole key) even without the window's highest reading,
|
||||
or when the median of the last hour (three readings or more) is over
|
||||
`flag_above` too. A lone slow reading opens nothing.
|
||||
`floor` is the smallest departure worth calling an anomaly, and
|
||||
`floor_setting` the Settings → Thresholds floor it comes from. Hours an
|
||||
anomaly covers are learned only once someone marks it normal.
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"""Host tools: the fleet, one host in depth, and a host's agent metrics."""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Callable
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
@@ -19,6 +21,18 @@ from .tickets import open_tickets_for
|
||||
|
||||
AGENT_SOURCE = "host_agent"
|
||||
|
||||
# Which of a host's sensors don't count as its temperature, and why: unwired
|
||||
# inputs read a constant or move against the CPU. The plugin that knows its
|
||||
# sensors registers the judge; with none, every sensor counts.
|
||||
_UNCOUNTED_SENSORS: Callable[[str, dict[str, float]], dict[str, str]] | None = None
|
||||
|
||||
|
||||
def judge_sensors(judge: Callable[[str, dict[str, float]], dict[str, str]]) -> None:
|
||||
"""`judge(host_name, {sensor: celsius})` → {sensor: why} for each sensor
|
||||
that is not a thermometer; get_host marks those `counted: false`."""
|
||||
global _UNCOUNTED_SENSORS
|
||||
_UNCOUNTED_SENSORS = judge
|
||||
|
||||
|
||||
async def _all_hosts(db) -> list[Host]:
|
||||
return list((await db.execute(select(Host).order_by(Host.name))).scalars())
|
||||
@@ -50,7 +64,7 @@ _PART_ORDER = {
|
||||
"interfaces": lambda r: -((r.get("net_rx_bps") or 0) + (r.get("net_tx_bps") or 0)),
|
||||
"disk_io": lambda r: -((r.get("disk_read_bps") or 0) + (r.get("disk_write_bps") or 0)),
|
||||
"drive_health": lambda r: (-(r.get("drive_failing") or 0), -(r.get("drive_bad_sectors") or 0)),
|
||||
"temperatures": lambda r: -(r.get("temp_c") or 0),
|
||||
"temperatures": lambda r: (not r.get("counted", True), -(r.get("temp_c") or 0)),
|
||||
"other": lambda r: r["part"],
|
||||
}
|
||||
|
||||
@@ -79,6 +93,12 @@ def agent_parts(parts: dict[str, dict], host_name: str, limit: int) -> dict:
|
||||
groups["mounts"].append({"mount": name, "part": name, **metrics})
|
||||
else:
|
||||
groups["other"].append({"part": name, **metrics})
|
||||
temps = {r["sensor"]: r["temp_c"] for r in groups["temperatures"] if r.get("temp_c") is not None}
|
||||
uncounted = _UNCOUNTED_SENSORS(host_name, temps) if _UNCOUNTED_SENSORS and temps else {}
|
||||
for r in groups["temperatures"]:
|
||||
r["counted"] = r["sensor"] not in uncounted
|
||||
if not r["counted"]:
|
||||
r["not_counted_because"] = uncounted[r["sensor"]]
|
||||
more = (f"Pass a larger `parts_limit` (max 200), or read one part with query_metrics("
|
||||
f'source_module="host_agent", resource="{host_name}:<part>").')
|
||||
out: dict = {}
|
||||
@@ -148,7 +168,9 @@ async def get_host(host: str, parts_limit: int = 20) -> dict:
|
||||
|
||||
The agent's values are `agent.metrics` (host-level) and its sub-resources
|
||||
grouped by kind: `mounts` (fullest first), `interfaces` and `disk_io`
|
||||
(busiest first), `temperatures` (hottest first), each with `items`,
|
||||
(busiest first), `temperatures` (the ones that count as the host's
|
||||
temperature hottest first, then those left out of it: `counted: false`
|
||||
with `not_counted_because`, likely nothing wired to them), each with `items`,
|
||||
`total` and `truncated`; `cpu_cores` summarises per-core CPU (count,
|
||||
min/avg/max, the three busiest). One part's history is query_metrics with
|
||||
source_module="host_agent" and resource="<host>:<part>", where `part` is
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""The cutoffs a host's readings are graded by, as Settings → Thresholds sets
|
||||
them: amber past `warn`, red past `crit`."""
|
||||
from __future__ import annotations
|
||||
|
||||
from steward.core.settings import DEFAULTS, get_all_settings, set_setting, to_thresholds_cfg
|
||||
from steward.mcp._context import mcp_app
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from ._audit import audit_mcp
|
||||
from ._shape import db_session
|
||||
|
||||
# Each graded reading, its unit, and what it measures.
|
||||
READINGS = {
|
||||
"cpu": ("%", "CPU use"),
|
||||
"mem": ("%", "memory used"),
|
||||
"disk": ("%", "fullest mount's space used"),
|
||||
"load": ("%", "1-minute load per core"),
|
||||
"temp": ("°C", "hottest real thermometer"),
|
||||
}
|
||||
|
||||
|
||||
def _view(settings: dict) -> dict:
|
||||
cfg = to_thresholds_cfg(settings)
|
||||
return {kind: {"warn": cfg[kind]["warn"], "crit": cfg[kind]["crit"], "unit": unit, "measures": what,
|
||||
"default": {"warn": DEFAULTS[f"thresholds.{kind}_warn"],
|
||||
"crit": DEFAULTS[f"thresholds.{kind}_crit"]}}
|
||||
for kind, (unit, what) in READINGS.items()}
|
||||
|
||||
|
||||
async def get_thresholds() -> dict:
|
||||
"""The cutoffs host readings are graded by, as Settings → Thresholds
|
||||
sets them: per reading (cpu, mem, disk, load, temp), `warn` (amber past
|
||||
it) and `crit` (red past it) in `unit`, with Steward's default for each.
|
||||
The host pages, the Overview and get_overview grade by these. Alert
|
||||
rules have their own cutoffs (list_alert_rules); these don't fire
|
||||
anything. set_thresholds (admin) changes them.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
return _view(await get_all_settings(db))
|
||||
|
||||
|
||||
async def set_thresholds(cutoffs: dict[str, dict[str, float]]) -> dict:
|
||||
"""Change the cutoffs host readings are graded by, as Settings →
|
||||
Thresholds does. Takes effect on the next page load; audited.
|
||||
|
||||
cutoffs: {reading: {"warn": n, "crit": n}} for any of get_thresholds'
|
||||
readings, e.g. {"temp": {"crit": 90}}. Either of warn and crit may be
|
||||
left out and keeps its value; warn must stay below crit. A reading not
|
||||
named is left as it is. Returns the cutoffs as they now stand.
|
||||
"""
|
||||
unknown = sorted(set(cutoffs or {}) - set(READINGS))
|
||||
if unknown:
|
||||
raise ValueError(f"No reading named {', '.join(unknown)}; they are {', '.join(READINGS)}.")
|
||||
if not cutoffs:
|
||||
raise ValueError("Nothing to change: pass cutoffs, e.g. {\"temp\": {\"crit\": 90}}.")
|
||||
async with db_session() as db:
|
||||
current = _view(await get_all_settings(db))
|
||||
changes: dict = {}
|
||||
for kind, given in cutoffs.items():
|
||||
extra = sorted(set(given or {}) - {"warn", "crit"})
|
||||
if extra or not given:
|
||||
raise ValueError(f"{kind} takes warn and/or crit, not {', '.join(extra) or 'nothing'}.")
|
||||
new = {}
|
||||
for edge in ("warn", "crit"):
|
||||
if edge not in given:
|
||||
new[edge] = current[kind][edge]
|
||||
continue
|
||||
try:
|
||||
value = float(given[edge])
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"{kind} {edge} must be a number.") from None
|
||||
if not value > 0:
|
||||
raise ValueError(f"{kind} {edge} must be more than 0.")
|
||||
new[edge] = int(value) if value.is_integer() else value
|
||||
changes[f"thresholds.{kind}_{edge}"] = new[edge]
|
||||
if new["warn"] >= new["crit"]:
|
||||
raise ValueError(f"{kind} warn ({new['warn']}) must be below its crit ({new['crit']}).")
|
||||
app = mcp_app()
|
||||
async with db_session() as db:
|
||||
async with db.begin():
|
||||
before = await get_all_settings(db)
|
||||
for key, value in changes.items():
|
||||
await set_setting(db, key, value)
|
||||
fresh = await get_all_settings(db)
|
||||
app.config["THRESHOLDS"] = to_thresholds_cfg(fresh)
|
||||
await audit_mcp("settings.saved", "settings", "thresholds",
|
||||
{"section": "thresholds",
|
||||
"changes": {k: {"from": before.get(k), "to": v} for k, v in changes.items()}})
|
||||
return _view(fresh)
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(get_thresholds, role=UserRole.viewer, read_only=True)
|
||||
register_tool(set_thresholds, role=UserRole.admin, read_only=False)
|
||||
@@ -128,7 +128,7 @@ async def detail(monitor_id: str):
|
||||
norm = (await monitor_svc.latency_norms(db, [m])).get(m.id)
|
||||
anomalies = await anom.views(db, await anom.list_anomalies(
|
||||
db, source=m.type, resource=m.name,
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=20), links=False)
|
||||
since=datetime.now(timezone.utc) - timedelta(days=30), limit=100), links=False)
|
||||
host = await db.get(Host, m.host_id) if m.host_id else None
|
||||
thresholds = current_app.config.get("THRESHOLDS", {})
|
||||
latest = st["latest"]
|
||||
|
||||
@@ -446,7 +446,9 @@ a.stat-tile:hover { border-color: var(--border-mid); background: var(--bg-elevat
|
||||
.host-health-summary { font-size: 0.92rem; }
|
||||
/* Its four tiles share the card's width (two a row on a phone); each sparkline
|
||||
stretches across its tile. */
|
||||
.host-health-tiles { grid-template-columns: repeat(4, minmax(0, 1fr)); }
|
||||
.host-health-tiles { grid-template-columns: repeat(var(--tiles, 4), minmax(0, 1fr)); }
|
||||
/* Which sensor the temperature tile reads, under its line; cut with … when long. */
|
||||
.stat-tile-sensor { display: block; font-size: 0.72rem; color: var(--text-muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.host-health-tiles .stat-tile { min-width: 0; }
|
||||
.host-health-tiles .stat-tile-hint { margin-top: 0.25rem; }
|
||||
@media (max-width: 640px) { .host-health-tiles { grid-template-columns: repeat(2, minmax(0, 1fr)); } }
|
||||
|
||||
@@ -1,14 +1,31 @@
|
||||
{# A resource's metric anomalies (#5740), for its own page: `anomalies` are
|
||||
anomalies.views() rows, to review first; `back` is this page's path, so a
|
||||
review returns here. Shown only when there is one. #}
|
||||
{% from "_anomaly.html" import anomaly_row with context %}
|
||||
{# A resource's metric anomalies, for its own page: one line saying how many
|
||||
wait for review and which are out of their normal now, with a button to
|
||||
Alerts → Anomalies narrowed to this resource, where they are reviewed.
|
||||
`anomalies` are anomalies.views() rows; `base_resource` is the page's
|
||||
resource (none for a page covering a whole source, like Traefik). Shown
|
||||
only while one waits for review or is going on. #}
|
||||
{% if anomalies %}
|
||||
{% set going = anomalies | selectattr("ongoing") | list %}
|
||||
<section class="card{% if going %} problem-card{% endif %}" aria-labelledby="resource-anomalies" style="margin-bottom:1rem;">
|
||||
<h2 id="resource-anomalies" class="section-title" style="margin-bottom:0.35rem;">
|
||||
{% if going %}Out of its normal now{% else %}Metric anomalies, last 30 days{% endif %}
|
||||
{% set waiting = anomalies | selectattr("anomaly.review", "equalto", "to_review") | list %}
|
||||
{% set going = anomalies | selectattr("ongoing") | rejectattr("anomaly.review", "equalto", "normal") | list %}
|
||||
{% if waiting or going %}
|
||||
{% set q = {"source": anomalies[0].anomaly.source_module} %}
|
||||
{% if base_resource %}{% set _ = q.update({"resource": base_resource}) %}{% endif %}
|
||||
<section class="card{% if going %} problem-card{% endif %}" aria-labelledby="resource-anomalies"
|
||||
style="margin-bottom:1rem;display:flex;align-items:center;gap:0.5rem 1rem;flex-wrap:wrap;">
|
||||
<h2 id="resource-anomalies" class="section-title" style="margin:0;">
|
||||
{% if going %}Out of its normal now{% else %}Metric anomalies{% endif %}
|
||||
</h2>
|
||||
{% for v in anomalies %}{{ anomaly_row(v, back=back, show_resource=v.anomaly.resource_name != base_resource) }}{% endfor %}
|
||||
<div style="font-size:0.78rem;margin-top:0.4rem;"><a href="/alerts/anomalies">All anomalies</a></div>
|
||||
<span style="font-size:0.85rem;">
|
||||
{%- if going -%}
|
||||
{%- for v in going -%}
|
||||
{%- if v.anomaly.resource_name != base_resource %}{{ v.anomaly.resource_name }} {% endif -%}
|
||||
{{ metric_label(v.anomaly.metric_name) }}{{ ", " if not loop.last }}
|
||||
{%- endfor -%}
|
||||
{%- if waiting %} · {% endif -%}
|
||||
{%- endif -%}
|
||||
{%- if waiting %}<strong>{{ waiting|length }}</strong> to review{% endif -%}
|
||||
</span>
|
||||
<a class="btn btn-sm" style="margin-left:auto;" href="/alerts/anomalies?{{ q|urlencode }}">Review</a>
|
||||
</section>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
@@ -11,10 +11,17 @@
|
||||
{% macro divided(items) %}
|
||||
{% for item in items %}
|
||||
{% if item.opens_group %}<h3 class="table-divider" style="font-size:0.78rem;font-weight:500;color:var(--text);background:var(--bg-elevated);padding:0.35rem 0.75rem;margin:0.75rem -0.75rem 0;border-top:1px solid var(--border-mid);border-bottom:1px solid var(--border-mid);">{{ item.group }}</h3>{% endif %}
|
||||
{{ anomaly_row(item.v, back="/alerts/anomalies") }}
|
||||
{{ anomaly_row(item.v, back=here|default("/alerts/anomalies"), show_resource=not resource or item.v.anomaly.resource_name != resource) }}
|
||||
{% endfor %}
|
||||
{% endmacro %}
|
||||
|
||||
{% if source or resource %}
|
||||
<p class="field-hint" style="margin:0.75rem 0 0;">
|
||||
Only {% if resource %}<strong>{{ resource }}</strong>{% if source %} ({{ source_label(source) }}){% endif %}{% else %}{{ source_label(source) }}{% endif %}.
|
||||
<a href="/alerts/anomalies">Show every anomaly</a>
|
||||
</p>
|
||||
{% endif %}
|
||||
|
||||
<div class="card" style="margin-top:1rem;">
|
||||
<h2 class="section-title" style="margin-bottom:0.35rem;">
|
||||
{% if to_review %}{{ to_review|length }} to review{% else %}Nothing to review{% endif %}
|
||||
|
||||
@@ -61,7 +61,7 @@
|
||||
<div style="display:flex;align-items:baseline;justify-content:space-between;margin-bottom:0.5rem;">
|
||||
<h3 class="section-title" style="margin-bottom:0;">Monitors</h3>
|
||||
{% if session.user_role in ['operator', 'admin'] %}
|
||||
<a href="/monitors/new?host_id={{ host.id }}" class="btn btn-ghost btn-sm" style="margin-left:auto;margin-right:0.75rem;">Add monitor</a>
|
||||
<a href="/monitors/new?host_id={{ host.id }}" class="btn btn-ghost btn-sm" style="margin-left:auto;margin-right:0.75rem;flex-shrink:0;" title="Add a monitor for this host">Add</a>
|
||||
{% endif %}
|
||||
{% if uptime %}
|
||||
<span style="font-size:0.8rem;color:var(--text-muted);">
|
||||
|
||||
@@ -118,7 +118,7 @@ for _t in (
|
||||
# A source not listed here is Monitoring.
|
||||
_SOURCE_AREAS = {
|
||||
"icmp": "network", "dns": "network", "tcp": "service", "http": "service",
|
||||
"docker": "service", "prometheus": "service",
|
||||
"docker": "service", "prometheus": "service", "homeassistant": "service",
|
||||
"traefik": "network", "unifi": "network",
|
||||
"snmp": "hardware",
|
||||
}
|
||||
@@ -154,6 +154,8 @@ _AUDIT_TYPES = (
|
||||
("account.time_zone", None), ("account.email", None),
|
||||
("host_agent.maintenance_plan", "steward.config"), ("host_agent.maintenance", None),
|
||||
("docker.prune.", None),
|
||||
# An update is on the timeline as itself, once it ends (homeassistant.timeline).
|
||||
("homeassistant.update", None),
|
||||
# A change recorded by hand is on the timeline as itself (timeline.changes).
|
||||
("timeline.change", None),
|
||||
# A ticket's opening, resolution and reopening are on the timeline as
|
||||
|
||||
@@ -88,6 +88,45 @@ async def test_a_quiet_page_says_nothing_waits(app):
|
||||
assert "Nothing to review" in page and "within 14 days" in page
|
||||
|
||||
|
||||
async def test_a_page_narrowed_to_one_resource_says_so_and_a_review_returns_to_it(app):
|
||||
here = "/alerts/anomalies?source=host_agent&resource=nas"
|
||||
async with app.test_request_context(here):
|
||||
session["user_id"], session["user_role"] = "u1", "operator"
|
||||
page = await render_template("alerts/anomalies.html", review_days=14, past=[],
|
||||
to_review=[{"v": _view(), "opens_group": True, "group": "Today"}],
|
||||
source="host_agent", resource="nas", here=here)
|
||||
assert "<strong>nas</strong>" in page and '<a href="/alerts/anomalies">Show every anomaly</a>' in page
|
||||
assert 'name="next" value="/alerts/anomalies?source=host_agent&resource=nas"' in page
|
||||
|
||||
|
||||
async def _card(app, anomalies, base_resource="nas"):
|
||||
async with app.test_request_context("/hosts/h1"):
|
||||
session["user_id"], session["user_role"] = "u1", "operator"
|
||||
return await render_template("_anomaly_card.html", anomalies=anomalies,
|
||||
back="/hosts/h1", base_resource=base_resource)
|
||||
|
||||
|
||||
async def test_a_resources_card_is_one_line_counting_what_waits_with_a_way_to_review(app):
|
||||
ended = NOW - timedelta(minutes=5)
|
||||
page = await _card(app, [_view(ended_at=ended), _view(id="a2", metric_name="load_pct", ended_at=ended),
|
||||
_view(id="a3", review="normal", ended_at=ended)])
|
||||
assert "<strong>2</strong> to review" in page
|
||||
assert 'href="/alerts/anomalies?source=host_agent&resource=nas"' in page
|
||||
assert "This is normal" not in page and "Out of its normal now" not in page # reviewed there
|
||||
|
||||
|
||||
async def test_a_card_names_what_is_out_of_its_normal_now(app):
|
||||
page = await _card(app, [_view(resource_name="nas:/var", metric_name="disk_used_pct")])
|
||||
assert "Out of its normal now" in page and "nas:/var" in page and "problem-card" in page
|
||||
|
||||
|
||||
async def test_a_card_with_nothing_waiting_or_going_is_not_shown(app):
|
||||
reviewed = _view(review="normal", ended_at=NOW - timedelta(minutes=5))
|
||||
assert "resource-anomalies" not in await _card(app, [reviewed])
|
||||
going_but_accepted = _view(review="normal") # the new normal being learned
|
||||
assert "resource-anomalies" not in await _card(app, [going_but_accepted])
|
||||
|
||||
|
||||
async def _widget(app, logged_in, anomalies, to_review):
|
||||
async with app.test_request_context("/status/widget"):
|
||||
if logged_in:
|
||||
|
||||
@@ -42,6 +42,8 @@ async def test_ansible_comes_before_the_docker_list(app, monkeypatch):
|
||||
# Monitors are added on their own form, filed under this host (#5277);
|
||||
# each row's own page has the labelled Edit/Pause/Delete.
|
||||
assert 'href="/monitors/new?host_id=h1"' in page
|
||||
# The card's header shares one line with the uptime: a one-word button keeps it whole.
|
||||
assert 'title="Add a monitor for this host">Add</a>' in page
|
||||
assert "row-actions" not in page and 'action="/monitors/add"' not in page
|
||||
|
||||
|
||||
@@ -70,7 +72,7 @@ async def test_the_agent_card_keeps_its_actions_under_manage(app):
|
||||
app, "host_agent", "panel.html", host=host, reg=reg, reporting=True, stale=False,
|
||||
target=target, ansible_available=True, managed_key_set=True)
|
||||
manage = page.index(">Manage agent</summary>")
|
||||
assert page.index("Full metrics →") < manage
|
||||
assert "Full metrics" not in page # it heads the vitals strip, at the top of the page
|
||||
for action in (">Update agent</button>", ">Rotate token</button>", ">Unregister agent</button>"):
|
||||
assert page.index(action) > manage
|
||||
assert ">Remove</button>" not in page
|
||||
|
||||
@@ -9,7 +9,7 @@ import steward
|
||||
from steward.alerts.service import CORE_METRICS
|
||||
|
||||
_REPO = pathlib.Path(steward.__file__).parent.parent
|
||||
_PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp", "prometheus")
|
||||
_PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp", "prometheus", "homeassistant")
|
||||
|
||||
|
||||
def catalog() -> dict[str, set[str]]:
|
||||
|
||||
@@ -140,6 +140,49 @@ def test_a_burst_past_its_usual_peak_is_an_anomaly(app, src):
|
||||
assert a.started_at == NOW - timedelta(minutes=15)
|
||||
|
||||
|
||||
def _window_of(app, src: str, res: str, readings: dict[int, float]):
|
||||
"""A detection pass at NOW over `readings` ({minutes before NOW: value}),
|
||||
against the normal that holds 10 and peaks at 20."""
|
||||
from sqlalchemy import select
|
||||
from steward.core.anomalies import detect
|
||||
from steward.models.metrics import PluginMetric, PluginMetricLatest
|
||||
from steward.models.norms import MetricAnomaly
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add_all(_norms(src, res))
|
||||
s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC,
|
||||
value=v, recorded_at=NOW - timedelta(minutes=m))
|
||||
for m, v in readings.items()])
|
||||
await s.merge(PluginMetricLatest(source_module=src, resource_name=res, metric_name=METRIC,
|
||||
value=readings[min(readings)], recorded_at=NOW))
|
||||
async with s.begin():
|
||||
await detect(s, NOW)
|
||||
return list((await s.execute(select(MetricAnomaly).where(MetricAnomaly.resource_name == res))).scalars())
|
||||
return asyncio.run(_go())
|
||||
|
||||
|
||||
def _every_3_minutes(value: float = 10.0) -> dict[int, float]:
|
||||
"""Six readings across the 15-minute window, as a router or monitor reports."""
|
||||
return {m: value for m in range(0, 15, 3)} | {14: value}
|
||||
|
||||
|
||||
def test_one_slow_reading_is_no_anomaly(app, src):
|
||||
# One 200 among five 10s averages 41.7, past the peak of 20; without it the window holds 10.
|
||||
assert _window_of(app, src, _res("one"), _every_3_minutes() | {6: 200.0}) == []
|
||||
|
||||
|
||||
def test_two_slow_readings_past_the_peak_are_an_anomaly(app, src):
|
||||
(a,) = _window_of(app, src, _res("two"), _every_3_minutes() | {3: 200.0, 6: 200.0})
|
||||
assert a.started_at == NOW - timedelta(minutes=15)
|
||||
|
||||
|
||||
def test_a_lone_reading_is_no_anomaly(app, src):
|
||||
# A quiet router's only request in the hour: neither a height nor a held level.
|
||||
assert _window_of(app, src, _res("lone"), {2: 500.0}) == []
|
||||
|
||||
|
||||
def test_a_burst_past_its_slots_peak_but_within_what_most_hours_reach_is_no_anomaly(app, src):
|
||||
assert _hour_of(app, src, _res("anywhere"), level=30.0, for_minutes=15, anywhere=40.0) == []
|
||||
|
||||
|
||||
@@ -121,6 +121,40 @@ def test_a_tasks_history_moves_onto_its_service_a_batch_at_a_time(app, monkeypat
|
||||
assert [sp.anomaly_id for sp in spans] == [row.id]
|
||||
|
||||
|
||||
def test_keys_with_raw_samples_are_taken_first_and_a_pass_finishes_many_small_keys(app):
|
||||
"""Taken from the end of old_keys: a chart and a normal read recent history,
|
||||
so a key with raw samples doesn't wait behind thousands that only have old
|
||||
hours. One pass moves every small key it has time for."""
|
||||
from sqlalchemy import select
|
||||
from steward.models.metrics import PluginMetricHourly, PluginMetricLatest
|
||||
rekey = _docker("rekey")
|
||||
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
aged = [f"nas/it{tag}_old.1.{c * 25}" for c in "bcd"]
|
||||
recent = f"nas/it{tag}_new.1.{TASK}"
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add_all([_hour(k, "cpu_pct", NOW, avg=1.0, top=2.0) for k in aged])
|
||||
s.add(PluginMetricLatest(source_module="docker", resource_name=recent, metric_name="cpu_pct",
|
||||
value=1.0, recorded_at=NOW))
|
||||
async with s.begin():
|
||||
found = await rekey.old_keys(s)
|
||||
await rekey.make_rekey_task(app).coro_factory()
|
||||
async with app.db_sessionmaker() as s:
|
||||
left = (await s.execute(select(PluginMetricHourly.resource_name).where(
|
||||
PluginMetricHourly.resource_name.in_(aged)))).scalars().all()
|
||||
moved = (await s.execute(select(PluginMetricHourly.value_avg).where(
|
||||
PluginMetricHourly.resource_name == f"swarm/it{tag}_old"))).scalars().all()
|
||||
return found, left, moved
|
||||
|
||||
found, left, moved = asyncio.run(_go())
|
||||
assert all(found.index(recent) > found.index(k) for k in aged)
|
||||
assert left == []
|
||||
assert moved == [1.0]
|
||||
|
||||
|
||||
def test_an_alert_rule_on_a_task_follows_it_to_its_service(app):
|
||||
from sqlalchemy import select
|
||||
from steward.models.alerts import AlertOperator, AlertRule
|
||||
|
||||
@@ -0,0 +1,827 @@
|
||||
"""Integration: Steward signs in to Home Assistant, reads it, and installs one
|
||||
update only after a newer backup, following it to its end. A fake Home Assistant
|
||||
answers over httpx's mock transport. Requires STEWARD_DATABASE_URL."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
ADDON = "update.example_update"
|
||||
BEFORE, AFTER = "2026-10-30T03:00:00+00:00", "2026-10-31T03:00:00+00:00"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
def _ha(sub):
|
||||
from steward.core.plugin_manager import get_loaded_plugin_modules
|
||||
loaded = get_loaded_plugin_modules().get("homeassistant")
|
||||
base = loaded.__name__ if loaded is not None else "plugins.homeassistant"
|
||||
return importlib.import_module(f"{base}.{sub}")
|
||||
|
||||
|
||||
class FakeHomeAssistant:
|
||||
"""Its entities, the actions it received, what it refuses, and its sign-ins."""
|
||||
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
self.tokens = set()
|
||||
self.refreshes = []
|
||||
self.revoked = []
|
||||
self.refuse = False
|
||||
self.down = False
|
||||
self.admin = True
|
||||
self.reject_install = None
|
||||
self.reject_reboot = None
|
||||
self.cut_off = False
|
||||
self.entities = {
|
||||
ADDON: self._update("on", "1.0"),
|
||||
"sensor.backup_last_successful_automatic_backup": {"state": BEFORE, "attributes": {}},
|
||||
"event.backup_automatic_backup": {"state": BEFORE, "attributes": {"event_type": "completed"}},
|
||||
}
|
||||
|
||||
def _update(self, state, installed):
|
||||
return {"state": state, "attributes": {"title": "Example", "installed_version": installed,
|
||||
"latest_version": "1.1", "in_progress": False,
|
||||
"entity_picture": "/api/hassio/addons/example/icon"}}
|
||||
|
||||
def backup_lands(self):
|
||||
self.entities["sensor.backup_last_successful_automatic_backup"]["state"] = AFTER
|
||||
|
||||
def update_lands(self):
|
||||
self.entities[ADDON] = self._update("off", "1.1")
|
||||
|
||||
def handler(self, request: httpx.Request) -> httpx.Response:
|
||||
if self.down:
|
||||
raise httpx.ConnectError("connection refused")
|
||||
path = request.url.path
|
||||
if path == "/auth/token":
|
||||
form = parse_qs(request.content.decode())
|
||||
self.refreshes.append(form)
|
||||
if self.refuse:
|
||||
return httpx.Response(400, json={"error": "invalid_grant"})
|
||||
return httpx.Response(200, json={"access_token": f"access-{len(self.refreshes)}", "expires_in": 1800,
|
||||
"refresh_token": "refresh-new", "token_type": "Bearer"})
|
||||
if path == "/auth/revoke":
|
||||
self.revoked.append(parse_qs(request.content.decode())["token"][0])
|
||||
return httpx.Response(200)
|
||||
self.tokens.add(request.headers.get("authorization"))
|
||||
if self.refuse:
|
||||
return httpx.Response(401)
|
||||
if path == "/api/template":
|
||||
return httpx.Response(200, text="1") if self.admin else httpx.Response(401)
|
||||
if path.startswith("/api/services/"):
|
||||
self.calls.append((path.removeprefix("/api/services/"), json.loads(request.content)))
|
||||
if self.cut_off:
|
||||
raise httpx.ReadTimeout("no answer yet")
|
||||
if self.reject_install and path.endswith("update/install"):
|
||||
return httpx.Response(500, json={"message": self.reject_install})
|
||||
if self.reject_reboot and path.endswith("hassio/host_reboot"):
|
||||
return httpx.Response(500, json={"message": self.reject_reboot})
|
||||
return httpx.Response(200, json=[])
|
||||
if path == "/api/states":
|
||||
return httpx.Response(200, json=[{"entity_id": k, **v} for k, v in self.entities.items()])
|
||||
entity = self.entities.get(path.removeprefix("/api/states/"))
|
||||
return httpx.Response(200, json=entity) if entity else httpx.Response(404)
|
||||
|
||||
@property
|
||||
def transport(self):
|
||||
return httpx.MockTransport(self.handler)
|
||||
|
||||
|
||||
BACKUP = ("backup/create_automatic", {})
|
||||
INSTALL = ("update/install", {"entity_id": ADDON})
|
||||
|
||||
|
||||
async def _clean(app, **connection):
|
||||
from steward.core.crypto import encrypt_secret
|
||||
m = _ha("models")
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(delete(m.HomeAssistantUpdate))
|
||||
await s.execute(delete(m.HomeAssistantConnection))
|
||||
await s.execute(delete(m.HomeAssistantSchedule))
|
||||
s.add(m.HomeAssistantConnection(**{
|
||||
"url": "http://ha.test:8123", "client_id": "http://steward.test/",
|
||||
"refresh_token": encrypt_secret(f"refresh-{uuid.uuid4().hex}"), "can_install": True, **connection}))
|
||||
|
||||
|
||||
async def _state(app):
|
||||
m = _ha("models")
|
||||
async with app.db_sessionmaker() as s:
|
||||
connection = (await s.execute(select(m.HomeAssistantConnection))).scalars().first()
|
||||
runs = (await s.execute(select(m.HomeAssistantUpdate))).scalars().all()
|
||||
return connection, runs
|
||||
|
||||
|
||||
def test_a_poll_stores_what_it_read_and_records_its_metrics(app):
|
||||
from steward.models.metrics import PluginMetricLatest
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
connection, _ = await _state(app)
|
||||
async with app.db_sessionmaker() as s:
|
||||
rows = (await s.execute(select(PluginMetricLatest).where(
|
||||
PluginMetricLatest.source_module == "homeassistant"))).scalars().all()
|
||||
return connection, {r.metric_name: r.value for r in rows}
|
||||
|
||||
connection, metrics = asyncio.run(go())
|
||||
assert ha.tokens == {"Bearer access-1"}
|
||||
assert connection.last_error is None and not connection.refused
|
||||
assert [u["entity_id"] for u in connection.updates] == [ADDON]
|
||||
assert connection.backup["last_ok"] == BEFORE
|
||||
assert metrics["is_up"] == 1.0 and metrics["updates_waiting"] == 1.0 and metrics["backup_age_hours"] >= 0
|
||||
|
||||
|
||||
def test_a_sign_in_lasts_until_its_access_token_nearly_runs_out_then_is_refreshed(app):
|
||||
client = _ha("client")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
connection = (await _state(app))[0]
|
||||
first = await client.access_token(connection, ha.transport)
|
||||
again = await client.access_token(connection, ha.transport)
|
||||
assert first == again and len(ha.refreshes) == 1
|
||||
for key, (token, _until) in list(client._access.items()):
|
||||
client._access[key] = (token, time.monotonic() - 1)
|
||||
return await client.access_token(connection, ha.transport)
|
||||
|
||||
assert asyncio.run(go()) == "access-2"
|
||||
[form] = ha.refreshes[:1]
|
||||
assert form["grant_type"] == ["refresh_token"] and form["client_id"] == ["http://steward.test/"]
|
||||
|
||||
|
||||
def test_a_refused_sign_in_is_asked_once_and_not_again(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.refuse = True
|
||||
seen = []
|
||||
real = ha.handler
|
||||
|
||||
def counted(request):
|
||||
seen.append(request.url.path)
|
||||
return real(request)
|
||||
transport = httpx.MockTransport(counted)
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=transport)
|
||||
await sched.poll(app, transport=transport)
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert seen == ["/auth/token"]
|
||||
assert connection.refused and "Sign in again" in connection.last_error
|
||||
|
||||
|
||||
def test_an_update_waits_for_a_newer_backup_then_installs_and_ends_on_the_timeline(app):
|
||||
from steward.models.timeline import TimelineEntry
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
assert ha.calls == [BACKUP]
|
||||
assert run.state == "backing_up" and run.from_version == "1.0" and run.to_version == "1.1"
|
||||
|
||||
# No newer backup yet: nothing is installed, however often it is asked.
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
assert ha.calls == [BACKUP]
|
||||
|
||||
ha.backup_lands()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
_, [row] = await _state(app)
|
||||
assert row.state == "installing" and row.backup_at is not None
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
assert ha.calls == [BACKUP, INSTALL]
|
||||
|
||||
# Home Assistant restarting is expected, and isn't the end of the update.
|
||||
ha.down = True
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
_, [row] = await _state(app)
|
||||
assert row.state == "installing"
|
||||
ha.down = False
|
||||
|
||||
ha.update_lands()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
connection, [row] = await _state(app)
|
||||
assert row.state == "done" and row.finished_at is not None
|
||||
assert connection.last_polled_at is None # the page shows the new version at the next tick
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
assert len(ha.calls) == 2
|
||||
async with app.db_sessionmaker() as s:
|
||||
entries = (await s.execute(select(TimelineEntry).where(
|
||||
TimelineEntry.ref_type == "homeassistant_update", TimelineEntry.ref_id == row.id))).scalars().all()
|
||||
return entries
|
||||
|
||||
[entry] = asyncio.run(go())
|
||||
assert entry.type == "homeassistant.updated" and entry.actor == "someone"
|
||||
assert "1.0 to 1.1" in entry.summary and entry.area == "service"
|
||||
|
||||
|
||||
def test_a_failed_backup_installs_nothing(app):
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
ha.entities["event.backup_automatic_backup"] = {
|
||||
"state": AFTER, "attributes": {"event_type": "failed", "failed_reason": "upload_failed"}}
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
[run] = asyncio.run(go())
|
||||
assert run.state == "failed" and "upload_failed" in run.message and "Nothing was installed" in run.message
|
||||
assert ha.calls == [BACKUP]
|
||||
|
||||
|
||||
def test_an_action_home_assistant_is_still_working_on_counts_as_sent(app):
|
||||
"""It answers when finished, and an update that restarts it never does."""
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.cut_off = True
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
assert run.state == "backing_up"
|
||||
ha.backup_lands()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
[run] = asyncio.run(go())
|
||||
assert run.state == "installing" and run.install_sent_at is not None
|
||||
assert ha.calls == [BACKUP, INSTALL]
|
||||
|
||||
|
||||
def test_an_install_home_assistant_rejects_fails_with_what_it_said(app):
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.reject_install = "Not enough free space"
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
ha.backup_lands()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
[run] = asyncio.run(go())
|
||||
assert run.state == "failed" and "Not enough free space" in run.message
|
||||
|
||||
|
||||
def test_an_install_that_couldnt_reach_home_assistant_is_asked_for_again(app):
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
# Now, as the install's deadline counts from the backup while it is unsent.
|
||||
ha.entities["sensor.backup_last_successful_automatic_backup"]["state"] = datetime.now(timezone.utc).isoformat()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
ha.down = True
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
assert (await _state(app))[1][0].install_sent_at is None
|
||||
ha.down = False
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
[run] = asyncio.run(go())
|
||||
assert run.install_sent_at is not None and ha.calls == [BACKUP, INSTALL]
|
||||
|
||||
|
||||
def test_an_install_past_its_deadline_fails_and_only_one_runs_at_a_time(app):
|
||||
from datetime import timedelta
|
||||
sched, upd, m = _ha("scheduler"), _ha("updates"), _ha("models")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
run = await upd.start_update(app, ADDON, "someone", transport=ha.transport)
|
||||
with pytest.raises(upd.CannotStart, match="under way"):
|
||||
await upd.start_update(app, ADDON, "someone else", transport=ha.transport)
|
||||
ha.backup_lands()
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
row = await s.get(m.HomeAssistantUpdate, run.id)
|
||||
row.install_sent_at -= timedelta(seconds=upd.INSTALL_DEADLINE_S["addon"] + 1)
|
||||
await upd.advance(app, transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
[run] = asyncio.run(go())
|
||||
assert run.state == "failed" and "Still not on 1.1" in run.message
|
||||
assert len(ha.calls) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entity,change,says", [
|
||||
("update.unknown", None, "isn't a Core, OS or add-on"),
|
||||
(ADDON, "no_longer_waiting", "no longer waiting"),
|
||||
(ADDON, "not_an_admin", "isn't an administrator"),
|
||||
])
|
||||
def test_nothing_starts_unless_the_update_can_run(app, entity, change, says):
|
||||
sched, upd = _ha("scheduler"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app, can_install=change != "not_an_admin")
|
||||
await sched.poll(app, transport=ha.transport)
|
||||
if change == "no_longer_waiting":
|
||||
ha.update_lands()
|
||||
with pytest.raises(upd.CannotStart, match=says):
|
||||
await upd.start_update(app, entity, "someone", transport=ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
assert asyncio.run(go()) == []
|
||||
assert ha.calls == []
|
||||
|
||||
|
||||
def _admin_client(app):
|
||||
from steward.models.users import User, UserRole
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
uid = str(uuid.uuid4())
|
||||
|
||||
async def make():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=f"ha-{tag}", email=f"ha-{tag}@test", password_hash="x",
|
||||
role=UserRole.admin, is_active=True))
|
||||
client = app.test_client()
|
||||
async with client.session_transaction() as sess:
|
||||
sess.update(user_id=uid, user_role="admin", username=f"ha-{tag}", _csrf="test-token", login_at=time.time())
|
||||
return client, f"ha-{tag}"
|
||||
return make()
|
||||
|
||||
|
||||
CSRF = {"X-CSRF-Token": "test-token"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("admin", [True, False])
|
||||
def test_signing_in_goes_to_home_assistant_and_keeps_what_comes_back(app, monkeypatch, admin):
|
||||
from steward.core.crypto import decrypt_secret
|
||||
client, m = _ha("client"), _ha("models")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.admin = admin
|
||||
for name in ("redeem_code", "is_admin", "revoke"):
|
||||
real = getattr(client, name)
|
||||
if name == "revoke":
|
||||
monkeypatch.setattr(client, name, lambda c, real=real: real(c, ha.transport))
|
||||
else:
|
||||
monkeypatch.setattr(client, name, lambda *a, real=real: real(*a, ha.transport))
|
||||
|
||||
async def go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(delete(m.HomeAssistantUpdate))
|
||||
await s.execute(delete(m.HomeAssistantConnection))
|
||||
web, username = await _admin_client(app)
|
||||
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
|
||||
form={"url": "http://ha.test:8123/", "verify_tls": "on"})
|
||||
assert response.status_code == 302
|
||||
to = urlparse(response.headers["Location"])
|
||||
query = parse_qs(to.query)
|
||||
assert f"{to.scheme}://{to.netloc}{to.path}" == "http://ha.test:8123/auth/authorize"
|
||||
assert query["response_type"] == ["code"]
|
||||
assert urlparse(query["redirect_uri"][0]).netloc == urlparse(query["client_id"][0]).netloc
|
||||
assert query["redirect_uri"][0].endswith("/plugins/homeassistant/connection/callback")
|
||||
|
||||
# Coming back with someone else's state changes nothing.
|
||||
forged = await web.get("/plugins/homeassistant/connection/callback?code=c&state=forged")
|
||||
assert forged.status_code == 400 and not ha.refreshes
|
||||
assert (await _state(app))[0] is None
|
||||
|
||||
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
|
||||
form={"url": "http://ha.test:8123/", "verify_tls": "on"})
|
||||
state = parse_qs(urlparse(response.headers["Location"]).query)["state"][0]
|
||||
back = await web.get(f"/plugins/homeassistant/connection/callback?code=the-code&state={state}")
|
||||
assert back.status_code == 302
|
||||
again = await web.get(f"/plugins/homeassistant/connection/callback?code=the-code&state={state}")
|
||||
assert again.status_code == 400 and len(ha.refreshes) == 1 # a state works once
|
||||
connection = (await _state(app))[0]
|
||||
|
||||
removed = await web.post("/plugins/homeassistant/connection/delete", headers=CSRF)
|
||||
assert removed.status_code == 302
|
||||
return connection, username, (await _state(app))[0]
|
||||
|
||||
connection, username, after = asyncio.run(go())
|
||||
assert ha.refreshes[0]["grant_type"] == ["authorization_code"] and ha.refreshes[0]["code"] == ["the-code"]
|
||||
assert connection.url == "http://ha.test:8123" and connection.verify_tls
|
||||
assert decrypt_secret(connection.refresh_token, context="test") == "refresh-new"
|
||||
assert connection.refresh_token != "refresh-new"
|
||||
assert connection.can_install is admin and connection.signed_in_by == username and connection.signed_in_at
|
||||
assert ha.revoked == ["refresh-new"] and after is None
|
||||
|
||||
|
||||
def test_signing_in_again_ends_the_sign_in_it_replaces(app, monkeypatch):
|
||||
from steward.core.crypto import encrypt_secret
|
||||
client = _ha("client")
|
||||
ha = FakeHomeAssistant()
|
||||
for name in ("redeem_code", "is_admin"):
|
||||
real = getattr(client, name)
|
||||
monkeypatch.setattr(client, name, lambda *a, real=real: real(*a, ha.transport))
|
||||
real_revoke = client.revoke
|
||||
monkeypatch.setattr(client, "revoke", lambda c: real_revoke(c, ha.transport))
|
||||
|
||||
async def go():
|
||||
await _clean(app, refresh_token=encrypt_secret("refresh-old"), refused=True)
|
||||
web, _ = await _admin_client(app)
|
||||
response = await web.post("/plugins/homeassistant/connection/sign-in", headers=CSRF,
|
||||
form={"url": "http://ha.test:8123", "verify_tls": "on"})
|
||||
state = parse_qs(urlparse(response.headers["Location"]).query)["state"][0]
|
||||
await web.get(f"/plugins/homeassistant/connection/callback?code=c&state={state}")
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert ha.revoked == ["refresh-old"] and not connection.refused and connection.last_polled_at is None
|
||||
|
||||
|
||||
# ── The system, over the websocket ───────────────────────────────────────────
|
||||
|
||||
class FakeSocket:
|
||||
"""Home Assistant's websocket: `answers` by question type or Supervisor endpoint,
|
||||
where one left out is refused as it is for a non-administrator."""
|
||||
|
||||
def __init__(self, answers, auth="auth_ok"):
|
||||
self.answers, self.auth = answers, auth
|
||||
self.waiting = [{"type": "auth_required"}]
|
||||
self.asked = []
|
||||
|
||||
def __call__(self, url, **options):
|
||||
self.url = url
|
||||
return self
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
return False
|
||||
|
||||
async def send(self, text):
|
||||
question = json.loads(text)
|
||||
if question["type"] == "auth":
|
||||
self.waiting.append({"type": self.auth})
|
||||
return
|
||||
key = question.get("endpoint") or question["type"]
|
||||
self.asked.append(key)
|
||||
# Something unasked arriving first must be passed over.
|
||||
self.waiting.append({"type": "event", "id": 9999})
|
||||
self.waiting.append({"id": question["id"], "success": key in self.answers, "result": self.answers.get(key)})
|
||||
|
||||
async def recv(self):
|
||||
return json.dumps(self.waiting.pop(0))
|
||||
|
||||
|
||||
def test_a_poll_keeps_the_system_problems_and_records_them(app):
|
||||
from steward.models.metrics import PluginMetricLatest
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
socket = FakeSocket({
|
||||
"get_config": {"recovery_mode": False},
|
||||
"repairs/list_issues": {"issues": [{"domain": "hassio", "translation_key": "issue_x", "severity": "warning"}]},
|
||||
"frontend/get_translations": {"resources": {"component.hassio.issues.issue_x.title": "Something's wrong"}},
|
||||
"/resolution/info": {"unhealthy": [], "unsupported": []},
|
||||
"/host/info": {"disk_free": 50.0, "disk_total": 100.0},
|
||||
"/addons": {"addons": [{"slug": "a", "state": "started"}, {"slug": "b", "name": "B", "state": "stopped"}]},
|
||||
"/addons/b/info": {"boot": "auto"},
|
||||
})
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=socket)
|
||||
connection, _ = await _state(app)
|
||||
async with app.db_sessionmaker() as s:
|
||||
rows = (await s.execute(select(PluginMetricLatest).where(
|
||||
PluginMetricLatest.source_module == "homeassistant"))).scalars().all()
|
||||
return connection, {r.metric_name: r.value for r in rows}
|
||||
|
||||
connection, metrics = asyncio.run(go())
|
||||
assert socket.url == "ws://ha.test:8123/api/websocket"
|
||||
assert "/addons/a/info" not in socket.asked # a running add-on isn't asked after
|
||||
assert [p["title"] for p in connection.health["problems"]] == ["Something's wrong", "B should start on its own and isn't running."]
|
||||
assert metrics["problems_open"] == 2.0 and metrics["disk_free_pct"] == 50.0 and metrics["update_failed"] == 0.0
|
||||
|
||||
|
||||
def test_the_page_shows_storage_used_of_the_total_with_the_percentage(app):
|
||||
async def go():
|
||||
await _clean(app, health={"repairs_read": True, "problems": [], "disk_free_gb": 99.8, "disk_used_gb": 12.0,
|
||||
"disk_total_gb": 116.5, "disk_free_pct": 85.7, "disk_used_pct": 10.3})
|
||||
web, _ = await _admin_client(app)
|
||||
return await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
|
||||
page = asyncio.run(go())
|
||||
assert "12.0 of 116.5 GB (10%)" in page
|
||||
assert "Disk free" not in page
|
||||
|
||||
|
||||
def test_the_page_says_install_leaves_the_reboot_to_you_and_the_schedule_makes_it(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = _ha_with_os_update_waiting()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
web, _ = await _admin_client(app)
|
||||
return await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
|
||||
page = asyncio.run(go())
|
||||
assert "It takes effect once you reboot the machine" in page
|
||||
assert "an OS update reboots the machine, with nobody there to watch" in page
|
||||
|
||||
|
||||
def test_a_report_it_couldnt_read_leaves_the_last_one_and_home_assistant_still_answers(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
def unreachable(url, **options):
|
||||
raise OSError("connection refused")
|
||||
|
||||
async def go():
|
||||
await _clean(app, health={"repairs_read": True, "problems": [{"title": "Old"}]})
|
||||
await sched.poll(app, transport=ha.transport, connect=unreachable)
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert connection.last_error is None and not connection.refused
|
||||
assert connection.health["problems"] == [{"title": "Old"}] and "connection refused" in connection.health["error"]
|
||||
|
||||
|
||||
def test_a_refused_websocket_stops_the_polling_like_any_refusal(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}, auth="auth_invalid"))
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert connection.refused and "Sign in again" in connection.last_error
|
||||
|
||||
|
||||
# ── Updates on a schedule ────────────────────────────────────────────────────
|
||||
|
||||
CORE = "update.home_assistant_core_update"
|
||||
|
||||
|
||||
async def _a_slot_just_passed(app, minutes=10):
|
||||
from datetime import timedelta
|
||||
m = _ha("models")
|
||||
at = datetime.now(timezone.utc) - timedelta(minutes=minutes)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(m.HomeAssistantSchedule(enabled=True, weekday=at.weekday(), at_time=f"{at:%H:%M}", time_zone="UTC",
|
||||
window_minutes=240, created_at=at - timedelta(days=30)))
|
||||
|
||||
|
||||
async def _schedule_row(app):
|
||||
m = _ha("models")
|
||||
async with app.db_sessionmaker() as s:
|
||||
return (await s.execute(select(m.HomeAssistantSchedule))).scalars().first()
|
||||
|
||||
|
||||
def test_a_slot_installs_the_add_on_first_then_core_and_stops_at_a_failure(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
m = _ha("models")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.entities[CORE] = ha._update("on", "1.0")
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport) # the slot fires and asks for a fresh read
|
||||
assert not (await _state(app))[1] and (await _state(app))[0].last_polled_at is None
|
||||
await plan.step(app, ha.transport) # nothing starts before that read
|
||||
assert not (await _state(app))[1]
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
[first] = (await _state(app))[1]
|
||||
assert first.entity_id == ADDON and first.started_by == "schedule"
|
||||
await plan.step(app, ha.transport) # one at a time
|
||||
assert len((await _state(app))[1]) == 1
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
run = await s.get(m.HomeAssistantUpdate, first.id)
|
||||
run.state, run.message = "failed", "It broke."
|
||||
await plan.step(app, ha.transport)
|
||||
assert "It broke." in (await _schedule_row(app)).stopped
|
||||
await plan.step(app, ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
assert [r.entity_id for r in asyncio.run(go())] == [ADDON]
|
||||
assert ha.calls == [BACKUP]
|
||||
|
||||
|
||||
def test_a_slot_moves_on_after_a_good_update_and_never_retries_one(app):
|
||||
sched, plan, updates = _ha("scheduler"), _ha("schedule"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.entities[CORE] = ha._update("on", "1.0")
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
ha.backup_lands()
|
||||
await updates.advance(app, ha.transport) # the backup landed
|
||||
await updates.advance(app, ha.transport) # the install is sent
|
||||
ha.update_lands()
|
||||
await updates.advance(app, ha.transport) # it's done, and asks for a fresh read
|
||||
await plan.step(app, ha.transport)
|
||||
assert len((await _state(app))[1]) == 1
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
runs = (await _state(app))[1]
|
||||
return {r.entity_id: r.state for r in runs}
|
||||
|
||||
assert asyncio.run(go()) == {ADDON: "done", CORE: "backing_up"}
|
||||
|
||||
|
||||
def test_a_slot_that_cant_start_an_update_says_why_and_asks_no_more(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
del ha.entities["sensor.backup_last_successful_automatic_backup"]
|
||||
await plan.step(app, ha.transport)
|
||||
return await _schedule_row(app)
|
||||
|
||||
row = asyncio.run(go())
|
||||
assert "wasn't started" in row.stopped and "no automatic backups" in row.stopped and not ha.calls
|
||||
|
||||
|
||||
def test_a_slot_past_its_window_starts_nothing_and_a_new_schedule_waits_for_its_next(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _a_slot_just_passed(app, minutes=300)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
assert not (await _state(app))[1]
|
||||
await _clean(app)
|
||||
at = datetime.now(timezone.utc)
|
||||
await plan.save(app, enabled=True, weekday=at.weekday(), at_time="00:00", time_zone="UTC")
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
assert asyncio.run(go()) == []
|
||||
assert ha.calls == []
|
||||
|
||||
|
||||
REBOOT = ("hassio/host_reboot", {})
|
||||
|
||||
|
||||
def _ha_with_os_update_waiting():
|
||||
ha = FakeHomeAssistant()
|
||||
del ha.entities[ADDON]
|
||||
ha.entities[_ha("service").OS] = ha._update("on", "1.0")
|
||||
return ha
|
||||
|
||||
|
||||
async def _scheduled_os_update_written(app, ha):
|
||||
"""A slot's OS update, followed to where it has been written and needs its reboot."""
|
||||
sched, plan, updates = _ha("scheduler"), _ha("schedule"), _ha("updates")
|
||||
await _clean(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
ha.backup_lands()
|
||||
await updates.advance(app, ha.transport) # the backup landed
|
||||
await updates.advance(app, ha.transport) # the install is sent
|
||||
# Home Assistant reports the new version as installed before the reboot.
|
||||
ha.entities[_ha("service").OS] = ha._update("off", "1.1")
|
||||
await updates.advance(app, ha.transport)
|
||||
[run] = (await _state(app))[1]
|
||||
assert run.state == "rebooting" and REBOOT not in ha.calls
|
||||
|
||||
|
||||
def test_a_scheduled_os_update_reboots_once_and_is_done_when_the_new_version_runs(app):
|
||||
updates = _ha("updates")
|
||||
ha = _ha_with_os_update_waiting()
|
||||
|
||||
def down(url, **options):
|
||||
raise OSError("connection refused")
|
||||
|
||||
async def go():
|
||||
await _scheduled_os_update_written(app, ha)
|
||||
await updates.advance(app, ha.transport)
|
||||
assert ha.calls.count(REBOOT) == 1
|
||||
|
||||
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.0", "version_pending": "1.1"}}))
|
||||
assert (await _state(app))[1][0].state == "rebooting" # not gone down yet
|
||||
web, _ = await _admin_client(app)
|
||||
page = await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
assert "rebooting into 1.1" in page and 'hx-trigger="every 15s"' in page
|
||||
|
||||
await updates.advance(app, ha.transport, down)
|
||||
assert (await _state(app))[1][0].state == "rebooting" # down, and expected to be
|
||||
await updates.advance(app, ha.transport, FakeSocket({}))
|
||||
assert (await _state(app))[1][0].state == "rebooting" # back, but the Supervisor isn't answering yet
|
||||
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.1", "version_pending": None}}))
|
||||
return (await _state(app))[1][0]
|
||||
|
||||
run = asyncio.run(go())
|
||||
assert run.state == "done" and run.finished_at is not None
|
||||
assert ha.calls.count(REBOOT) == 1
|
||||
|
||||
|
||||
def test_a_machine_that_falls_back_to_the_old_os_fails_at_the_deadline_and_ends_the_slot(app):
|
||||
from datetime import timedelta
|
||||
updates, plan, m = _ha("updates"), _ha("schedule"), _ha("models")
|
||||
ha = _ha_with_os_update_waiting()
|
||||
old_os = FakeSocket({"/os/info": {"version": "1.0", "version_pending": None}})
|
||||
|
||||
async def go():
|
||||
await _scheduled_os_update_written(app, ha)
|
||||
await updates.advance(app, ha.transport)
|
||||
await updates.advance(app, ha.transport, old_os)
|
||||
assert (await _state(app))[1][0].state == "rebooting"
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
run = (await s.execute(select(m.HomeAssistantUpdate))).scalars().first()
|
||||
run.reboot_sent_at = datetime.now(timezone.utc) - timedelta(minutes=updates.REBOOT_DEADLINE_S // 60 + 1)
|
||||
await updates.advance(app, ha.transport, FakeSocket({"/os/info": {"version": "1.0", "version_pending": None}}))
|
||||
await _ha("scheduler").poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
return (await _state(app))[1][0], await _schedule_row(app)
|
||||
|
||||
run, schedule = asyncio.run(go())
|
||||
assert run.state == "failed" and "isn't running" in run.message
|
||||
assert "isn't running" in schedule.stopped
|
||||
|
||||
|
||||
def test_a_refused_reboot_fails_the_update_and_says_it_still_needs_one(app):
|
||||
updates = _ha("updates")
|
||||
ha = _ha_with_os_update_waiting()
|
||||
ha.reject_reboot = "Not now"
|
||||
|
||||
async def go():
|
||||
await _scheduled_os_update_written(app, ha)
|
||||
await updates.advance(app, ha.transport)
|
||||
return (await _state(app))[1][0]
|
||||
|
||||
run = asyncio.run(go())
|
||||
assert run.state == "failed" and "Not now" in run.message and "needs a reboot" in run.message
|
||||
|
||||
|
||||
def test_an_os_update_started_by_a_person_leaves_the_reboot_to_them(app):
|
||||
sched, updates = _ha("scheduler"), _ha("updates")
|
||||
ha = _ha_with_os_update_waiting()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await updates.start_update(app, _ha("service").OS, "someone", ha.transport)
|
||||
ha.backup_lands()
|
||||
await updates.advance(app, ha.transport)
|
||||
await updates.advance(app, ha.transport)
|
||||
ha.entities[_ha("service").OS] = ha._update("off", "1.1")
|
||||
await updates.advance(app, ha.transport)
|
||||
await updates.advance(app, ha.transport)
|
||||
return (await _state(app))[1][0]
|
||||
|
||||
assert asyncio.run(go()).state == "done"
|
||||
assert REBOOT not in ha.calls
|
||||
@@ -82,7 +82,7 @@ def test_an_invite_is_emailed_and_the_invitee_makes_their_own_account_once(app,
|
||||
assert resp.status_code == 302 and resp.headers["Location"].endswith("?done=invited")
|
||||
assert len(sent) == 1 and sent[0]["To"] == to
|
||||
text = sent[0].get_body(("plain",)).get_content()
|
||||
assert f"{aname} invited you to Steward" in text and "EDT" in text and "UTC" in text
|
||||
assert f"{aname} invited you to Steward" in text and re.search(r"\bE[DS]T\b", text) and "UTC" in text
|
||||
key = _KEY.search(text).group(1)
|
||||
|
||||
page = await (await admin.get(resp.headers["Location"])).get_data(as_text=True)
|
||||
|
||||
@@ -134,3 +134,38 @@ def test_a_scrape_records_series_rates_and_a_down_target_fires_its_rule(app):
|
||||
assert rows[down].last_ok_at is None
|
||||
assert down_latest[(down, "is_up")] == 0.0
|
||||
assert state.state == AlertStateEnum.firing
|
||||
|
||||
|
||||
def test_a_refused_token_is_asked_once(app):
|
||||
"""Home Assistant bans an address after a few refusals, so a refused token
|
||||
isn't retried at each interval, however long ago the last scrape was."""
|
||||
from sqlalchemy import select
|
||||
|
||||
scheduler = _prom("scheduler")
|
||||
PrometheusTarget = _prom("models").PrometheusTarget
|
||||
TokenRefused = _prom("scraper").TokenRefused
|
||||
name = f"refused-{uuid.uuid4().hex[:8]}"
|
||||
asked = []
|
||||
|
||||
async def fake_scrape(url, token, verify_tls):
|
||||
asked.append(url)
|
||||
raise TokenRefused("Answered 401: the token was refused.")
|
||||
|
||||
async def go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(PrometheusTarget(name=name, url=f"http://{name}/metrics"))
|
||||
await scheduler.scrape_due(app, scrape=fake_scrape)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
row = (await s.execute(select(PrometheusTarget).where(PrometheusTarget.name == name))).scalar_one()
|
||||
refused = row.refused
|
||||
row.last_scraped_at = row.last_scraped_at.replace(year=2000)
|
||||
await scheduler.scrape_due(app, scrape=fake_scrape)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.delete((await s.execute(select(PrometheusTarget).where(PrometheusTarget.name == name))).scalar_one())
|
||||
return refused
|
||||
|
||||
assert asyncio.run(go()) is True
|
||||
assert asked == [f"http://{name}/metrics"]
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
"""get_thresholds / set_thresholds: the cutoffs host readings are graded by,
|
||||
read and changed as Settings → Thresholds does."""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from types import SimpleNamespace as NS
|
||||
|
||||
import pytest
|
||||
|
||||
from steward.mcp.tools import thresholds
|
||||
from steward.models.users import UserRole
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(monkeypatch):
|
||||
"""The settings table, the app and the audit log, in memory."""
|
||||
saved: dict = {"thresholds.temp_crit": 85}
|
||||
app = NS(config={})
|
||||
audited: list = []
|
||||
|
||||
class _Db:
|
||||
@asynccontextmanager
|
||||
async def begin(self):
|
||||
yield
|
||||
|
||||
@asynccontextmanager
|
||||
async def db_session():
|
||||
yield _Db()
|
||||
|
||||
async def get_all_settings(_db):
|
||||
return dict(saved)
|
||||
|
||||
async def set_setting(_db, key, value):
|
||||
saved[key] = value
|
||||
|
||||
async def audit_mcp(*args):
|
||||
audited.append(args)
|
||||
|
||||
monkeypatch.setattr(thresholds, "db_session", db_session)
|
||||
monkeypatch.setattr(thresholds, "get_all_settings", get_all_settings)
|
||||
monkeypatch.setattr(thresholds, "set_setting", set_setting)
|
||||
monkeypatch.setattr(thresholds, "mcp_app", lambda: app)
|
||||
monkeypatch.setattr(thresholds, "audit_mcp", audit_mcp)
|
||||
return NS(saved=saved, app=app, audited=audited)
|
||||
|
||||
|
||||
async def test_the_cutoffs_read_with_their_units_and_defaults(store):
|
||||
got = await thresholds.get_thresholds()
|
||||
assert set(got) == {"cpu", "mem", "disk", "load", "temp"}
|
||||
assert got["temp"] == {"warn": 70, "crit": 85, "unit": "°C", "measures": "hottest real thermometer",
|
||||
"default": {"warn": 70, "crit": 85}}
|
||||
|
||||
|
||||
async def test_raising_one_cutoff_keeps_the_other_and_grades_by_it_at_once(store):
|
||||
got = await thresholds.set_thresholds({"temp": {"crit": 90}})
|
||||
assert got["temp"]["crit"] == 90 and got["temp"]["warn"] == 70
|
||||
assert store.saved == {"thresholds.temp_crit": 90} # warn isn't written
|
||||
assert store.app.config["THRESHOLDS"]["temp"]["crit"] == 90
|
||||
(_action, _kind, _id, detail), = store.audited
|
||||
assert detail["changes"] == {"thresholds.temp_crit": {"from": 85, "to": 90}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cutoffs, refusal", [
|
||||
({}, "Nothing to change"),
|
||||
({"fan": {"crit": 1}}, "No reading named fan"),
|
||||
({"temp": {"high": 90}}, "warn and/or crit"),
|
||||
({"temp": {}}, "warn and/or crit"),
|
||||
({"temp": {"crit": "hot"}}, "must be a number"),
|
||||
({"temp": {"crit": 0}}, "more than 0"),
|
||||
({"temp": {"crit": 60}}, r"warn \(70\) must be below its crit \(60\)"),
|
||||
])
|
||||
async def test_refusals_change_nothing(store, cutoffs, refusal):
|
||||
with pytest.raises(ValueError, match=refusal):
|
||||
await thresholds.set_thresholds(cutoffs)
|
||||
assert store.saved == {"thresholds.temp_crit": 85} and not store.audited
|
||||
|
||||
|
||||
def test_the_tools_are_registered_with_their_roles(app):
|
||||
mcp = app.mcp_server
|
||||
for name, role, read_only in (("get_thresholds", UserRole.viewer, True),
|
||||
("set_thresholds", UserRole.admin, False)):
|
||||
assert mcp.tool_roles[name] is role, name
|
||||
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is read_only, name
|
||||
+16
-1
@@ -312,10 +312,25 @@ def test_get_host_groups_parts_by_kind_and_summarises_cores():
|
||||
assert got["disk_io"]["items"][0]["device"] == "sda"
|
||||
assert [d["device"] for d in got["drive_health"]["items"]] == ["nvme0", "sda"] # failing first
|
||||
assert got["temperatures"]["items"][0] == {"sensor": "Package id 0", "part": "temp:Package id 0",
|
||||
"temp_c": 61.0}
|
||||
"temp_c": 61.0, "counted": True}
|
||||
assert "other" not in got
|
||||
|
||||
|
||||
def test_get_host_marks_the_sensors_the_hosts_temperature_leaves_out(monkeypatch):
|
||||
"""An unwired input reading 115 °C is listed after the real sensors, with
|
||||
why it doesn't count, so a reader doesn't take it for heat."""
|
||||
from steward.mcp.tools import hosts
|
||||
monkeypatch.setattr(hosts, "_UNCOUNTED_SENSORS",
|
||||
lambda host, temps: {k: "reads a value no wired sensor gives"
|
||||
for k, c in temps.items() if c >= 100})
|
||||
got = hosts.agent_parts({"temp:SYSTIN": {"temp_c": 115.0}, "temp:Package id 0": {"temp_c": 61.0},
|
||||
"temp:nvme": {"temp_c": 40.0}}, "box", limit=10)["temperatures"]["items"]
|
||||
assert [t["sensor"] for t in got] == ["Package id 0", "nvme", "SYSTIN"]
|
||||
assert got[0]["counted"] is True and "not_counted_because" not in got[0]
|
||||
assert got[2]["counted"] is False
|
||||
assert got[2]["not_counted_because"] == "reads a value no wired sensor gives"
|
||||
|
||||
|
||||
def test_one_silent_agent_is_one_finding():
|
||||
"""#5721: VDNT-Docker03's silent agent read as three findings. Its agent-
|
||||
offline alert speaks for it, the stale-agent finding goes, and its rules
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Reading what Home Assistant reports: which updates Steward may install, how
|
||||
its backups stand, and what the pages and MCP show."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace as NS
|
||||
|
||||
import plugins.homeassistant as homeassistant_plugin
|
||||
from plugins.homeassistant import mcp_tools, schedule, scheduler, service as s
|
||||
from steward.models.users import UserRole
|
||||
from tests._templates import render_plugin_template
|
||||
from tests.test_style_tokens import hex_colours
|
||||
|
||||
NOW = datetime(2026, 10, 31, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _entity(entity_id, state="on", **attrs):
|
||||
return {"entity_id": entity_id, "state": state, "attributes": attrs}
|
||||
|
||||
|
||||
def _addon(slug="core_example", state="on"):
|
||||
return _entity(f"update.{slug}_update", state, title="Example add-on", installed_version="1.0",
|
||||
latest_version="1.1", entity_picture=f"/api/hassio/addons/{slug}/icon")
|
||||
|
||||
|
||||
def test_only_core_os_and_add_ons_can_be_installed():
|
||||
assert s.classify(_entity(s.CORE)) == "core"
|
||||
assert s.classify(_entity(s.OS)) == "os"
|
||||
assert s.classify(_addon()) == "addon"
|
||||
assert s.classify(_entity(s.SUPERVISOR, entity_picture="/api/brands/integration/hassio/icon.png")) is None
|
||||
assert s.classify(_entity("update.switch_firmware", device_class="firmware",
|
||||
entity_picture="/api/brands/integration/unifi/icon.png")) is None
|
||||
assert s.classify(_entity("update.custom_locks_update",
|
||||
entity_picture="https://brands.home-assistant.io/_/locks/icon.png")) is None
|
||||
|
||||
|
||||
def test_a_snapshot_keeps_waiting_updates_first():
|
||||
snapshot = s.read_snapshot([
|
||||
_addon("core_quiet", "off"), _entity("update.switch_firmware", title="Switch"),
|
||||
_addon(), _entity(s.CORE, title="Home Assistant Core"),
|
||||
_entity("update.gone", "unavailable"),
|
||||
_entity(s.LAST_BACKUP, "2026-10-31T03:00:00+00:00"), _entity(s.NEXT_BACKUP, "unknown"),
|
||||
_entity(s.BACKUP_EVENT, "2026-10-31T03:00:00+00:00", event_type="completed"),
|
||||
_entity("light.kitchen", "on"),
|
||||
])
|
||||
assert [u["entity_id"] for u in snapshot["updates"]] == [
|
||||
s.CORE, "update.core_example_update", "update.switch_firmware", "update.core_quiet_update"]
|
||||
backup = snapshot["backup"]
|
||||
assert backup["last_ok"] == "2026-10-31T03:00:00+00:00" and backup["next"] is None
|
||||
assert backup["last_result"] == "completed"
|
||||
|
||||
|
||||
def test_a_release_link_is_only_kept_when_it_is_a_web_address():
|
||||
assert s.read_snapshot([_entity(s.CORE, release_url="javascript:alert(1)")])["updates"][0]["release_url"] is None
|
||||
assert s.read_snapshot([_entity(s.CORE, release_url="https://example.com/r")])["updates"][0]["release_url"]
|
||||
|
||||
|
||||
def test_newer_is_judged_on_the_readings_alone():
|
||||
assert s.is_newer("2026-10-31T03:00:01+00:00", "2026-10-31T03:00:00+00:00")
|
||||
assert not s.is_newer("2026-10-31T03:00:00+00:00", "2026-10-31T03:00:00+00:00")
|
||||
assert not s.is_newer("2026-10-30T03:00:00+00:00", "2026-10-31T03:00:00+00:00")
|
||||
assert s.is_newer("2026-10-31T03:00:00+00:00", "unknown")
|
||||
assert not s.is_newer("unavailable", "2026-10-31T03:00:00+00:00")
|
||||
|
||||
|
||||
def test_how_the_backups_read():
|
||||
ok = {"known": True, "last_ok": (NOW - timedelta(hours=25)).isoformat(), "last_result": "completed"}
|
||||
assert s.backup_verdict(ok, NOW)[0] == "ok"
|
||||
assert abs(s.backup_age_hours(ok, NOW) - 25) < 0.01
|
||||
stale = {**ok, "last_ok": (NOW - timedelta(hours=27)).isoformat()}
|
||||
assert s.backup_verdict(stale, NOW) == ("warn", "The last good backup is 27 hours old, past the 26 hours expected.")
|
||||
failed = {**ok, "last_result": "failed", "failed_reason": "upload_failed"}
|
||||
assert "failed (upload_failed)" in s.backup_verdict(failed, NOW)[1]
|
||||
assert "no successful" in s.backup_verdict({"known": True, "last_ok": None}, NOW)[1]
|
||||
assert "reports no automatic backups" in s.backup_verdict({}, NOW)[1]
|
||||
|
||||
|
||||
def test_a_refused_sign_in_stops_polling_until_someone_acts():
|
||||
fresh = NS(refused=False, last_polled_at=NOW - timedelta(seconds=60))
|
||||
assert not scheduler.is_due(fresh, NOW)
|
||||
assert scheduler.is_due(NS(refused=False, last_polled_at=NOW - timedelta(seconds=300)), NOW)
|
||||
assert scheduler.is_due(NS(refused=False, last_polled_at=None), NOW)
|
||||
assert not scheduler.is_due(NS(refused=True, last_polled_at=None), NOW)
|
||||
|
||||
|
||||
def _connection(**over):
|
||||
return NS(**{"url": "http://ha.example:8123", "refresh_token": "tok-s3cret", "can_install": True,
|
||||
"verify_tls": True, "signed_in_by": "someone", "signed_in_at": NOW,
|
||||
"refused": False, "last_error": None, "last_polled_at": NOW, "last_ok_at": NOW,
|
||||
"backup": {"known": True, "last_ok": NOW.isoformat(), "last_result": "completed"},
|
||||
"updates": [], "health": {}, **over})
|
||||
|
||||
|
||||
def test_the_overview_names_what_needs_a_look_and_no_more():
|
||||
assert mcp_tools.overview_findings(None, None, NOW) == []
|
||||
assert mcp_tools.overview_findings(_connection(), None, NOW) == []
|
||||
waiting = _connection(updates=[{"waiting": True, "kind": "core", "title": "Core"}])
|
||||
assert mcp_tools.overview_findings(waiting, None, NOW) == []
|
||||
down = mcp_tools.overview_findings(_connection(last_error="Couldn't reach Home Assistant."), None, NOW)
|
||||
assert [f["kind"] for f in down] == ["home_assistant_down"]
|
||||
stale = _connection(backup={"known": True, "last_ok": (NOW - timedelta(days=2)).isoformat()})
|
||||
assert [f["kind"] for f in mcp_tools.overview_findings(stale, None, NOW)] == ["home_assistant_backup"]
|
||||
failed = NS(state="failed", title="Core", message="No new backup.", finished_at=NOW)
|
||||
[f] = mcp_tools.overview_findings(_connection(), failed, NOW)
|
||||
assert f["kind"] == "home_assistant_update_failed" and "No new backup." in f["summary"]
|
||||
two = {"problems": [{"title": "Disk trouble"}, {"title": "Another"}]}
|
||||
[f] = mcp_tools.overview_findings(_connection(health=two), None, NOW)
|
||||
assert f["kind"] == "home_assistant_problem" and f["summary"] == "Disk trouble (and 1 more)"
|
||||
plan = NS(stopped="Core: it failed.", last_slot_at=None)
|
||||
[f] = mcp_tools.overview_findings(_connection(), None, NOW, plan)
|
||||
assert f["kind"] == "home_assistant_schedule_stopped" and "Core: it failed." in f["summary"]
|
||||
|
||||
|
||||
def test_system_problems_are_read_from_what_home_assistant_answers():
|
||||
healthy = s.read_health({"config": {"recovery_mode": False}, "issues": [], "resolution": {"unhealthy": [], "unsupported": []},
|
||||
"host": {"disk_free": 99.8, "disk_used": 12.0, "disk_total": 116.5},
|
||||
"addons": [{"slug": "a", "state": "started"}]})
|
||||
assert healthy["problems"] == [] and healthy["repairs_read"] and healthy["supervised"]
|
||||
assert healthy["disk_free_pct"] == 85.7 and healthy["disk_used_gb"] == 12.0 and healthy["disk_used_pct"] == 10.3
|
||||
bad = s.read_health({
|
||||
"config": {"recovery_mode": True},
|
||||
"issues": [{"domain": "hassio", "translation_key": "issue_system_reboot_required", "severity": "warning"},
|
||||
{"domain": "demo", "translation_key": "named", "translation_placeholders": {"who": "X"}, "severity": "critical"},
|
||||
{"domain": "demo", "translation_key": "gone", "ignored": True},
|
||||
{"domain": "demo", "translation_key": "old", "dismissed_version": "1.0"}],
|
||||
"titles": {"component.demo.issues.named.title": "{who} broke"},
|
||||
"resolution": {"unhealthy": ["docker_gateway"], "unsupported": []},
|
||||
"host": {"disk_free": 5.0, "disk_total": 100.0},
|
||||
"addons": [{"slug": "should", "name": "Should run", "state": "stopped"}, {"slug": "once", "state": "stopped"}],
|
||||
"boots": {"should": "auto", "once": "manual"}})
|
||||
assert [p["kind"] for p in bad["problems"]] == ["recovery", "repair", "repair", "supervisor", "disk", "addon"]
|
||||
titles = [p["title"] for p in bad["problems"]]
|
||||
assert titles[1] == "System reboot required" and titles[2] == "X broke" and "docker gateway" in titles[3]
|
||||
assert "5 GB of 100 GB" in titles[4] and "Should run" in titles[5]
|
||||
|
||||
|
||||
def test_what_home_assistant_wouldnt_answer_is_not_read_as_fine():
|
||||
found = s.read_health({"config": {}, "issues": None, "resolution": None, "host": None, "addons": None})
|
||||
assert not found["repairs_read"] and not found["supervised"]
|
||||
assert found["disk_free_pct"] is None and found["disk_used_pct"] is None
|
||||
|
||||
|
||||
def test_a_schedule_installs_add_ons_then_core_then_the_os():
|
||||
def update(entity_id, kind, in_progress=False):
|
||||
return {"entity_id": entity_id, "kind": kind, "in_progress": in_progress}
|
||||
waiting = [update("os", "os"), update("core", "core"), update("addon", "addon")]
|
||||
assert schedule.next_to_install(waiting, set())["entity_id"] == "addon"
|
||||
assert schedule.next_to_install(waiting, {"addon"})["entity_id"] == "core"
|
||||
assert schedule.next_to_install(waiting, {"addon", "core"})["entity_id"] == "os"
|
||||
assert schedule.next_to_install(waiting, {"addon", "core", "os"}) is None
|
||||
assert schedule.next_to_install([update("addon", "addon", in_progress=True)], set()) is None
|
||||
|
||||
|
||||
def test_reading_is_open_to_a_viewer_and_installing_is_for_an_admin():
|
||||
seen = {}
|
||||
|
||||
def register(fn, *, role, read_only): # both required, as on the server
|
||||
seen[fn.__name__] = (role, read_only)
|
||||
homeassistant_plugin.register_mcp_tools(register)
|
||||
assert seen == {"get_home_assistant": (UserRole.viewer, True), "update_home_assistant": (UserRole.admin, False),
|
||||
"set_home_assistant_schedule": (UserRole.admin, False)}
|
||||
|
||||
|
||||
def _page_values(**over):
|
||||
update = {"entity_id": s.CORE, "title": "Core", "kind": "core", "kind_word": "Core", "installed": "1.0",
|
||||
"latest": "1.1", "waiting": True, "in_progress": False, "auto_update": False,
|
||||
"release_url": "https://example.com/r"}
|
||||
return {"connection": _connection(), "reach": "up", "says": "Answering.", "installable": [update], "others": [],
|
||||
"current": [], "runs": [], "under_way": None, "backup": {}, "backup_tone": "ok",
|
||||
"backup_says": "Backups are up to date.", "backup_at": NOW, "backup_next": None, "backup_age": 1.0,
|
||||
"stale_hours": 26, "health": {"repairs_read": True, "supervised": True}, "problems": [], "plan": None,
|
||||
"plan_when": None, "plan_next": None, "weekdays": ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
|
||||
"Saturday", "Sunday"), "default_zone": "UTC", **over}
|
||||
|
||||
|
||||
async def _render(app, role, **values):
|
||||
from quart import session
|
||||
async with app.test_request_context("/plugins/homeassistant/"):
|
||||
session["user_role"] = role
|
||||
page = await render_plugin_template(app, "homeassistant", "homeassistant/index.html", **values)
|
||||
return " ".join(page.split())
|
||||
|
||||
|
||||
async def test_an_admin_is_asked_before_an_install_and_a_viewer_cant_install(app):
|
||||
page = await _render(app, "admin", **_page_values())
|
||||
assert "1 update waiting" in page and 'action="/plugins/homeassistant/install"' in page
|
||||
assert "Steward takes a backup first" in page and "Home Assistant restarts." in page
|
||||
assert "tok-s3cret" not in page
|
||||
main = page[page.index("<main"):]
|
||||
assert not hex_colours(main), hex_colours(main)
|
||||
assert 'action="/plugins/homeassistant/install"' not in await _render(app, "viewer", **_page_values())
|
||||
|
||||
|
||||
async def test_install_is_held_back_while_it_couldnt_work_and_the_page_says_why(app):
|
||||
run = NS(id="r", title="Core", state="backing_up", from_version="1.0", to_version="1.1", started_by="someone",
|
||||
started_at=NOW, message=None)
|
||||
busy = await _render(app, "admin", **_page_values(under_way=run, runs=[run]))
|
||||
assert 'action="/plugins/homeassistant/install"' not in busy and "taking a backup first" in busy
|
||||
assert 'hx-trigger="every 15s"' in busy
|
||||
status_only = await _render(app, "admin", **_page_values(connection=_connection(can_install=False)))
|
||||
assert 'action="/plugins/homeassistant/install"' not in status_only
|
||||
assert "isn't an administrator" in status_only
|
||||
|
||||
|
||||
async def test_the_page_shows_system_problems_and_what_it_couldnt_read(app):
|
||||
clear = await _render(app, "viewer", **_page_values())
|
||||
assert "No repairs open" in clear
|
||||
problem = {"kind": "repair", "severity": "warning", "title": "System reboot required"}
|
||||
shown = await _render(app, "viewer", **_page_values(problems=[problem]))
|
||||
assert "System reboot required" in shown and "Settings → System → Repairs" in shown
|
||||
hidden = await _render(app, "viewer", **_page_values(health={"repairs_read": False, "supervised": False}))
|
||||
assert "wouldn't share its repairs" in hidden
|
||||
bare = await _render(app, "viewer", **_page_values(health={"repairs_read": True, "supervised": False}))
|
||||
assert "No Supervisor answered" in bare
|
||||
|
||||
|
||||
async def test_an_admin_sets_the_schedule_warned_of_what_it_does_and_a_viewer_only_reads_it(app):
|
||||
off = await _render(app, "admin", **_page_values())
|
||||
assert 'action="/plugins/homeassistant/schedule"' in off and "Off. Updates are only installed" in off
|
||||
assert "with nobody there to watch" in off
|
||||
assert '<option value="5" selected>Saturday</option>' in off and 'value="05:00"' in off
|
||||
plan = NS(enabled=True, weekday=5, at_time="05:00", time_zone="Europe/London", stopped="Core: it failed.")
|
||||
values = _page_values(plan=plan, plan_when="Saturdays at 05:00 (Europe/London), up to 4 h", plan_next=NOW)
|
||||
on = await _render(app, "viewer", **values)
|
||||
assert "Saturdays at 05:00" in on and "Core: it failed." in on
|
||||
assert 'action="/plugins/homeassistant/schedule"' not in on
|
||||
|
||||
|
||||
async def test_an_unconnected_page_offers_to_connect(app):
|
||||
page = await _render(app, "admin", connection=None)
|
||||
assert 'href="/plugins/homeassistant/connection"' in page
|
||||
|
||||
|
||||
async def _form(app, **values):
|
||||
from quart import session
|
||||
async with app.test_request_context("/plugins/homeassistant/connection"):
|
||||
session["user_role"] = "admin"
|
||||
page = await render_plugin_template(app, "homeassistant", "homeassistant/form.html", error=None,
|
||||
url="http://ha.example:8123", verify_tls=True, **values)
|
||||
return " ".join(page.split())
|
||||
|
||||
|
||||
async def test_the_form_warns_what_the_credential_can_do_and_recommends_a_service_account(app):
|
||||
page = await _form(app, connection=None)
|
||||
assert "same access as the user who signs in" in page and "including controlling every device" in page
|
||||
assert "We recommend a service account" in page and "in Home Assistant's logs" in page
|
||||
assert 'action="/plugins/homeassistant/connection/sign-in"' in page and "Continue to Home Assistant" in page
|
||||
assert "connection/delete" not in page
|
||||
|
||||
|
||||
async def test_a_connected_form_says_who_signed_in_and_never_shows_the_credential(app):
|
||||
page = await _form(app, connection=_connection())
|
||||
assert "tok-s3cret" not in page and "by someone" in page and "an administrator" in page
|
||||
assert "Sign in again" in page and "connection/delete" in page
|
||||
@@ -146,3 +146,19 @@ def test_build_sample_first_call_has_no_rates_but_keeps_state(monkeypatch):
|
||||
assert s1["net"] == [] and s1["diskio"] == [] # no prior counters yet
|
||||
assert state["mono"] is not None
|
||||
assert state["net"] == {"eth0": (1000, 500)}
|
||||
|
||||
|
||||
def test_collect_temps_says_which_chip_each_sensor_is_on(tmp_path, monkeypatch):
|
||||
nvme = tmp_path / "hwmon1"
|
||||
nvme.mkdir()
|
||||
(nvme / "name").write_text("nvme\n")
|
||||
(nvme / "temp3_input").write_text("49850\n")
|
||||
(nvme / "temp3_label").write_text("Sensor 2\n")
|
||||
bare = tmp_path / "hwmon2"
|
||||
bare.mkdir()
|
||||
(bare / "name").write_text("acpitz\n")
|
||||
(bare / "temp1_input").write_text("27800\n")
|
||||
monkeypatch.setattr(agent, "HWMON_DIR", str(tmp_path))
|
||||
got = sorted(agent.collect_temps(), key=lambda t: t["label"])
|
||||
assert got == [{"label": "Sensor 2", "celsius": 49.9, "chip": "nvme"},
|
||||
{"label": "acpitz_temp1", "celsius": 27.8, "chip": "acpitz"}]
|
||||
|
||||
@@ -114,6 +114,24 @@ def test_the_heat_alert_names_the_sensor_that_set_it():
|
||||
assert hottest_sensor("box", {"AUXTIN0": 110.0}) == []
|
||||
|
||||
|
||||
def test_the_hottest_sensor_is_named_by_its_chip_once_the_agent_says_it():
|
||||
"""An NVMe drive's "Sensor 2" alone doesn't say it's the drive."""
|
||||
from plugins.host_agent.routes import hottest_sensor
|
||||
sample = {"ts": "2026-06-16T00:00:00+00:00",
|
||||
"temps": [{"label": "Sensor 2", "celsius": 49.0, "chip": "nvme"},
|
||||
{"label": "Package id 0", "celsius": 41.0, "chip": "coretemp"},
|
||||
{"label": "nct6798_temp9", "celsius": 30.0, "chip": "nct6798"},
|
||||
{"label": "fan_ic", "celsius": 35.0, "chip": "mystery9"}]}
|
||||
idx = _index(_expand_sample_to_metrics(sample, "chipbox", datetime.now(timezone.utc)))
|
||||
assert idx[("temp_c", "chipbox:temp:Sensor 2")] == 49.0 # keyed by label alone
|
||||
temps = {"Sensor 2": 49.0, "Package id 0": 41.0}
|
||||
assert hottest_sensor("chipbox", temps) == ["NVMe · Sensor 2 at 49°C"]
|
||||
assert hottest_sensor("chipbox", {"Package id 0": 41.0}) == ["CPU · Package id 0 at 41°C"]
|
||||
assert hottest_sensor("chipbox", {"nct6798_temp9": 30.0}) == ["nct6798_temp9 at 30°C"]
|
||||
assert hottest_sensor("chipbox", {"fan_ic": 35.0}) == ["mystery9 · fan_ic at 35°C"]
|
||||
assert hottest_sensor("otherbox", temps) == ["Sensor 2 at 49°C"] # an agent that never said
|
||||
|
||||
|
||||
def test_a_wired_board_input_still_counts():
|
||||
"""#5759: below 100 °C a board input is a real reading."""
|
||||
from plugins.host_agent.routes import counts_as_hottest
|
||||
|
||||
@@ -25,16 +25,16 @@ def test_worst_reading(readings, worst):
|
||||
assert worst_reading(readings, TH) == worst
|
||||
|
||||
|
||||
async def _vitals(app, *, cpu=40.0, mem=50.0, stale=False, seen_min=1):
|
||||
async def _vitals(app, *, cpu=40.0, mem=50.0, stale=False, seen_min=1, temp=None, hottest=None):
|
||||
readings = (("CPU", cpu, "cpu"), ("Memory", mem, "mem"), ("Disk /", 30.0, "disk"),
|
||||
("Load per core", 20, "load"))
|
||||
("Load per core", 20, "load"), ("Temperature", temp, "temp"))
|
||||
async with app.test_request_context("/plugins/host_agent/vitals/h1"):
|
||||
page = await render_plugin_template(
|
||||
app, "host_agent", "_host_vitals.html", reporting=True, stale=stale,
|
||||
reg=SimpleNamespace(last_seen_at=datetime.now(timezone.utc) - timedelta(minutes=seen_min),
|
||||
agent_version="1.9.0"),
|
||||
cpu=cpu, mem=mem, disk_root=30.0, load_per_core=20,
|
||||
sparks={"cpu": "", "mem": "", "disk": "", "load": ""},
|
||||
reg=SimpleNamespace(host_id="h1", agent_version="1.9.0",
|
||||
last_seen_at=datetime.now(timezone.utc) - timedelta(minutes=seen_min)),
|
||||
cpu=cpu, mem=mem, disk_root=30.0, load_per_core=20, temp=temp, hottest=hottest,
|
||||
sparks={"cpu": "", "mem": "", "disk": "", "load": "", "temp": "<svg></svg>"},
|
||||
psi={"cpu": None, "mem": None, "io": None}, worst=worst_reading(readings, TH))
|
||||
return re.sub(r">\s+<", "><", " ".join(page.split()))
|
||||
|
||||
@@ -53,6 +53,28 @@ async def test_a_quiet_host_says_so(app):
|
||||
assert "Every reading is within its thresholds." in await _vitals(app)
|
||||
|
||||
|
||||
async def test_the_strip_at_the_top_of_the_page_leads_to_the_full_metrics(app):
|
||||
"""The temperatures, charts and every mount are on the full-metrics page;
|
||||
its link sat in the Agent card below the fold and was missed."""
|
||||
assert 'href="/plugins/host_agent/h1/"' in await _vitals(app)
|
||||
|
||||
|
||||
async def test_the_strip_shows_the_hottest_sensor_with_its_name_and_line(app):
|
||||
page = await _vitals(app, temp=62.0, hottest="Package id 0")
|
||||
assert "--tiles:5;" in page
|
||||
assert ('class="stat-tile stat-ok"><span class="stat-tile-label">Temperature</span>'
|
||||
'<span class="stat-tile-value">62°C</span>') in page
|
||||
assert '<svg></svg><span class="stat-tile-sensor" title="Package id 0">Package id 0</span>' in page
|
||||
without = await _vitals(app)
|
||||
assert "--tiles:4;" in without and ">Temperature<" not in without
|
||||
|
||||
|
||||
async def test_a_host_past_its_heat_threshold_says_so_in_degrees(app):
|
||||
page = await _vitals(app, temp=88.0, hottest="Package id 0") # crit at 85
|
||||
assert '<span class="tone-crit">Temperature 88°C</span> is past its critical threshold.' in page
|
||||
assert 'class="stat-tile stat-crit"><span class="stat-tile-label">Temperature</span>' in page
|
||||
|
||||
|
||||
async def test_a_stale_host_says_since_when_before_any_reading(app):
|
||||
page = await _vitals(app, mem=93.0, stale=True, seen_min=20)
|
||||
assert '<span class="badge badge-warn">Stale</span>' in page
|
||||
|
||||
@@ -270,7 +270,7 @@ def test_a_zypper_update_that_asks_for_a_reboot_still_worked(maint, monkeypatch)
|
||||
monkeypatch.setattr(maint, "_os_release_id", lambda path="/etc/os-release": "sles")
|
||||
ran = []
|
||||
|
||||
def fake_run(cmd, deadline=0, env=None, ok_codes=(0,)):
|
||||
def fake_run(cmd, deadline=0, env=None, ok_codes=(0,), keep=0):
|
||||
ran.append(cmd[2])
|
||||
code = 102 if cmd[2] == "update" else 0
|
||||
return code in ok_codes, "3 packages to upgrade.\n"
|
||||
@@ -282,6 +282,20 @@ def test_a_zypper_update_that_asks_for_a_reboot_still_worked(maint, monkeypatch)
|
||||
assert ran == ["refresh", "update"]
|
||||
|
||||
|
||||
def test_a_large_update_still_counts_its_packages(maint, monkeypatch):
|
||||
"""apt prints its count before the log of unpacking and setting up each
|
||||
package, which runs far past what a detail keeps."""
|
||||
_manager(monkeypatch, maint, ("apt-get",))
|
||||
log = "17 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n" + \
|
||||
"Setting up libfoo (1.2-3) ...\n" * 200
|
||||
monkeypatch.setattr(maint, "run", lambda cmd, **kw: (True, log if kw.get("keep", 0) is None
|
||||
else log[-maint.OUTPUT_KEPT_CHARS:]))
|
||||
monkeypatch.setattr(maint, "reboot_required", lambda: False)
|
||||
got = maint.do_update()
|
||||
assert got["ok"] and got["packages"] == 17
|
||||
assert len(got["detail"]) <= 500
|
||||
|
||||
|
||||
def test_a_failed_step_says_which(maint, monkeypatch):
|
||||
_manager(monkeypatch, maint, ("pacman",))
|
||||
monkeypatch.setattr(maint, "run", lambda cmd, **kw: (False, "error: failed to synchronize all databases"))
|
||||
|
||||
@@ -15,7 +15,7 @@ def _target(name, *, enabled=True, scraped=timedelta(seconds=30), error=None,
|
||||
url="http://archives:8080/metrics"):
|
||||
return NS(id=f"id-{name}", name=name, url=url, token="bearer-s3cret", enabled=enabled,
|
||||
interval_seconds=60, last_scraped_at=NOW - scraped if scraped else None,
|
||||
last_ok_at=NOW - timedelta(hours=1), last_error=error, last_kept=120, last_dropped=30,
|
||||
last_ok_at=NOW - timedelta(hours=1), last_error=error, refused=False, last_kept=120, last_dropped=30,
|
||||
last_capped=0, series_cap=1000, keep=["archives_*"], drop=[])
|
||||
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ import pytest
|
||||
|
||||
from plugins.prometheus.models import PrometheusTarget
|
||||
from plugins.prometheus.scheduler import is_due
|
||||
from plugins.prometheus.scraper import ScrapeError, scrape
|
||||
from plugins.prometheus.scraper import ScrapeError, TokenRefused, scrape
|
||||
from plugins.prometheus.service import (FormError, fold_states, label_pairs, map_families, parse_form,
|
||||
patterns, rate_name, resource_for, target_state, wanted)
|
||||
from steward.core import prometheus
|
||||
@@ -165,6 +165,16 @@ def test_state_and_due():
|
||||
assert target_state(t, now).reach == "paused" and not is_due(t, now)
|
||||
|
||||
|
||||
def test_a_refused_token_stops_the_scraping_and_the_page_says_so():
|
||||
now = datetime(2026, 10, 12, 12, 0, tzinfo=timezone.utc)
|
||||
t = PrometheusTarget(name="a", url="http://a", enabled=True, interval_seconds=60, refused=True,
|
||||
last_error="Answered 401: the token was refused.",
|
||||
last_scraped_at=now - timedelta(hours=5))
|
||||
state = target_state(t, now)
|
||||
assert state.reach == "down" and "stopped asking" in state.says and "Answered 401" in state.says
|
||||
assert not is_due(t, now)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_scrape_sends_the_token_and_says_why_it_failed():
|
||||
seen = {}
|
||||
@@ -180,7 +190,7 @@ async def test_a_scrape_sends_the_token_and_says_why_it_failed():
|
||||
transport = httpx.MockTransport(handler)
|
||||
fams = await scrape("http://svc/metrics", "tok", transport=transport)
|
||||
assert list(fams) == ["up_thing"] and seen["auth"] == "Bearer tok"
|
||||
with pytest.raises(ScrapeError, match="token was refused"):
|
||||
with pytest.raises(TokenRefused, match="token was refused"):
|
||||
await scrape("http://svc/denied", "bad", transport=transport)
|
||||
with pytest.raises(ScrapeError, match="no metrics"):
|
||||
await scrape("http://svc/html", transport=transport)
|
||||
|
||||
@@ -27,6 +27,7 @@ TOP_LEVEL = {
|
||||
"plugins/traefik/templates/traefik/index.html",
|
||||
"plugins/unifi/templates/unifi/index.html",
|
||||
"plugins/syslog/templates/syslog/index.html",
|
||||
"plugins/homeassistant/templates/homeassistant/index.html",
|
||||
# Logged out, or no page to go back up from.
|
||||
"steward/templates/auth/login.html",
|
||||
"steward/templates/auth/setup.html",
|
||||
|
||||
Reference in New Issue
Block a user