feat(homeassistant): it shows system problems and installs updates on a weekly slot, and leaves device readings alone (#6748, #6749, #6747)
The connection's job is keeping Home Assistant running. Each poll now reads open
repairs, Supervisor health, disk, recovery mode and add-ons that should be running
over its websocket, for the page, the overview and the problems_open, disk_free_pct
and update_failed metrics. That needs a websocket client, so websockets joins the
dependencies.
A weekly slot, set on the page or through MCP and off until then, installs add-ons,
then Core, then the OS, each after its own confirmed backup, ending at the first
failure. The slot arithmetic moves from the host agent's cadence to steward.core.weekly
so both share it.
The device scrape goes: the page's Metrics section and a target's token_from from
4c9ed02 are removed, and a migration deletes targets using it, as they'd otherwise ask
with no token. A refused token still stops a scrape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -1,6 +1,6 @@
|
||||
# Home Assistant plugin
|
||||
|
||||
Shows which Home Assistant updates are waiting and how old its last backup is, and installs Core, OS and add-on updates, each only after a fresh backup is confirmed. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API.
|
||||
Keeps a Home Assistant instance running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates, on a click or a weekly schedule, each only after a fresh backup is confirmed. It looks after Home Assistant itself and doesn't read your devices. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API.
|
||||
|
||||
## Setting it up
|
||||
|
||||
@@ -12,7 +12,7 @@ Your browser and Steward both need to reach Home Assistant at that address. Home
|
||||
|
||||
## What Steward holds
|
||||
|
||||
**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status and metrics, take backups and install updates. It is stored encrypted.
|
||||
**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates. It is stored encrypted.
|
||||
|
||||
We recommend a service account, a user made just for Steward, rather than your own. What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.
|
||||
|
||||
@@ -29,10 +29,22 @@ Install asks once, then Steward:
|
||||
|
||||
One runs at a time, and it carries on if Steward restarts. Firmware and custom integrations are left to Home Assistant, and Supervisor updates itself.
|
||||
|
||||
## System problems
|
||||
|
||||
Each read also asks Home Assistant whether the system itself has a problem, and the page and the overview show what it says:
|
||||
|
||||
- open repairs, from Settings → System → Repairs
|
||||
- the Supervisor calling the system unhealthy or unsupported
|
||||
- less than 10% of the disk free
|
||||
- starting in recovery mode, when its configuration didn't load
|
||||
- an add-on set to start on its own that isn't running
|
||||
|
||||
Repairs need an administrator's sign-in. Disk space and add-ons come from the Supervisor, so an install without one doesn't report them, and the page says so.
|
||||
|
||||
## Scheduled updates
|
||||
|
||||
Set a day and time under Scheduled updates on the page. It's off until you do. At that time Steward installs what's waiting, add-ons first, then Core, then the OS, one at a time and each after its own confirmed backup. It starts nothing new more than 4 hours after the slot. The first failure ends the run and the rest wait, and the page, the overview and `update_failed` say so. A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch, and an update can change how automations behave.
|
||||
|
||||
## Metrics
|
||||
|
||||
One sign-in covers everything: the Metrics section of the page reads Home Assistant's entity readings with the same user, through a target on the Prometheus page. **Read its metrics** makes that target, or **Use this sign-in** moves an existing one at that address onto it, keeping its name, filters and history and forgetting its token. Home Assistant only serves these with its Prometheus integration turned on (`prometheus:` in its `configuration.yaml`, then a restart). Steward doesn't change that for you; it says so if they aren't there.
|
||||
|
||||
If Home Assistant refuses that user, everything stops after one refusal, and signing in again or Try again starts everything. To go back to a separate token, type one into the target's form on the Prometheus page.
|
||||
|
||||
The poll itself records these under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting` and `backup_age_hours`. An alert rule on `backup_age_hours` above 26 warns when a daily backup has been missed; a rule's no-data warning covers Home Assistant going quiet.
|
||||
Under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting`, `backup_age_hours`, `problems_open`, `disk_free_pct` and `update_failed`. Useful alert rules: `backup_age_hours` above 26 for a missed daily backup, `problems_open` above 0, and `update_failed` above 0. A rule's no-data warning covers Home Assistant going quiet.
|
||||
|
||||
@@ -48,8 +48,6 @@ A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as
|
||||
|
||||
**A refused token stops the scraping.** When an endpoint answers 401 or 403, Steward stops asking and says so on the target's page and in the overview, because asking again won't change the answer and some services, Home Assistant among them, ban an address after a few refusals. Saving the target, usually with a new token, or pausing and resuming it, starts it again. Any other failure is retried at the next scrape.
|
||||
|
||||
**A target can use a plugin's sign-in instead of a token.** Home Assistant's page sets this up, so one Home Assistant user covers its metrics too. A token typed into the target's form replaces it. That sign-in is only sent to the address it was made for, and a refusal stops it from the plugin's page, not here.
|
||||
|
||||
A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state.
|
||||
|
||||
**Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.
|
||||
|
||||
@@ -11,7 +11,7 @@ _app: "Quart | None" = None
|
||||
def setup(app: "Quart") -> None:
|
||||
global _app
|
||||
_app = app
|
||||
from .models import HomeAssistantConnection, HomeAssistantUpdate # noqa: F401 — registers the tables
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate # noqa: F401 — registers the tables
|
||||
from .timeline import register
|
||||
register()
|
||||
|
||||
@@ -32,7 +32,8 @@ def get_nav() -> list[dict]:
|
||||
|
||||
def get_alert_metrics() -> dict[str, list[str]]:
|
||||
"""What the poll records, all under the resource "homeassistant"."""
|
||||
return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting"]}
|
||||
return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting", "problems_open", "disk_free_pct",
|
||||
"update_failed"]}
|
||||
|
||||
|
||||
def register_mcp_tools(register) -> None:
|
||||
|
||||
@@ -1,17 +1,20 @@
|
||||
# plugins/homeassistant/client.py
|
||||
"""Talking to Home Assistant, with what its own sign-in gave Steward: a refresh
|
||||
token kept in the connection, and access tokens made from it that last half an
|
||||
hour. Home Assistant bans an address after a few refused requests, so nothing is
|
||||
hour. Repairs and the Supervisor's reports are only offered on its websocket. Home Assistant bans an address after a few refused requests, so nothing is
|
||||
sent without a token and a refusal stops the asking. Every failure comes back as
|
||||
one sentence a page can show."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import ssl
|
||||
import time
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from websockets.asyncio.client import connect as ws_connect
|
||||
from websockets.exceptions import WebSocketException
|
||||
|
||||
from steward.core.crypto import decrypt_secret
|
||||
|
||||
@@ -23,6 +26,8 @@ ACTION_READ_SECONDS = 10.0
|
||||
MAX_BYTES = 16 * 1024 * 1024
|
||||
# An access token this close to running out isn't used for another request.
|
||||
EXPIRY_MARGIN_SECONDS = 60
|
||||
# One read of the system over the websocket, however its answers dribble in.
|
||||
SYSTEM_READ_SECONDS = 30
|
||||
|
||||
_access: dict[str, tuple[str, float]] = {} # refresh token -> access token, when to stop using it
|
||||
|
||||
@@ -118,12 +123,6 @@ async def redeem_code(url: str, client_id: str, code: str, verify_tls: bool, tra
|
||||
return refresh_token, _remember(refresh_token, found)
|
||||
|
||||
|
||||
def forget_access() -> None:
|
||||
"""Drop access tokens made so far: after a refusal the next ask starts from the
|
||||
sign-in, which a dead user is refused at without counting toward a ban."""
|
||||
_access.clear()
|
||||
|
||||
|
||||
async def access_token(connection, transport=None) -> str:
|
||||
refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in")
|
||||
token, until = _access.get(refresh_token, (None, 0.0))
|
||||
@@ -214,3 +213,65 @@ async def call_action(url: str, domain: str, action: str, data: dict, token: str
|
||||
except (ValueError, AttributeError):
|
||||
said = None
|
||||
raise Rejected(said or f"Home Assistant answered {status}.")
|
||||
|
||||
|
||||
# ── The system, over the websocket ───────────────────────────────────────────
|
||||
|
||||
class _Socket:
|
||||
"""Home Assistant's websocket after sign-in: one question at a time, each with
|
||||
an id, and None for one it won't answer (no Supervisor, or not an administrator)."""
|
||||
|
||||
def __init__(self, ws):
|
||||
self.ws, self.asked = ws, 0
|
||||
|
||||
async def sign_in(self, token: str) -> None:
|
||||
await self.ws.recv()
|
||||
await self.ws.send(json.dumps({"type": "auth", "access_token": token}))
|
||||
if json.loads(await self.ws.recv()).get("type") != "auth_ok":
|
||||
raise Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.")
|
||||
|
||||
async def ask(self, **question):
|
||||
self.asked += 1
|
||||
await self.ws.send(json.dumps({"id": self.asked, **question}))
|
||||
while True:
|
||||
answer = json.loads(await self.ws.recv())
|
||||
if answer.get("id") == self.asked:
|
||||
return answer.get("result") if answer.get("success") else None
|
||||
|
||||
async def supervisor(self, endpoint: str):
|
||||
return await self.ask(type="supervisor/api", endpoint=endpoint, method="get")
|
||||
|
||||
|
||||
async def read_system(url: str, token: str, verify_tls: bool, connect=None) -> dict:
|
||||
"""What service.read_health needs, as Home Assistant answered it. `connect`
|
||||
stands in for the websocket in tests."""
|
||||
target = "ws" + url.rstrip("/").removeprefix("http") + "/api/websocket"
|
||||
options: dict = {"max_size": MAX_BYTES}
|
||||
if target.startswith("wss:") and not verify_tls:
|
||||
options["ssl"] = ssl.create_default_context()
|
||||
options["ssl"].check_hostname, options["ssl"].verify_mode = False, ssl.CERT_NONE
|
||||
try:
|
||||
async with asyncio.timeout(SYSTEM_READ_SECONDS):
|
||||
async with (connect or ws_connect)(target, **options) as ws:
|
||||
socket = _Socket(ws)
|
||||
await socket.sign_in(token)
|
||||
found = {"config": await socket.ask(type="get_config"),
|
||||
"issues": (await socket.ask(type="repairs/list_issues") or {}).get("issues"),
|
||||
"resolution": await socket.supervisor("/resolution/info"),
|
||||
"host": await socket.supervisor("/host/info"), "titles": {}, "boots": {}}
|
||||
domains = sorted({i["domain"] for i in found["issues"] or []})
|
||||
if domains:
|
||||
titles = await socket.ask(type="frontend/get_translations", language="en",
|
||||
category="issues", integration=domains)
|
||||
found["titles"] = (titles or {}).get("resources") or {}
|
||||
found["addons"] = (await socket.supervisor("/addons") or {}).get("addons")
|
||||
# Whether one should be running is only on its own page, so a running one isn't asked.
|
||||
for addon in found["addons"] or []:
|
||||
if addon.get("state") != "started":
|
||||
info = await socket.supervisor(f"/addons/{addon['slug']}/info") or {}
|
||||
found["boots"][addon["slug"]] = info.get("boot")
|
||||
return found
|
||||
except TimeoutError:
|
||||
raise Unreachable(f"Home Assistant's system report didn't arrive within {SYSTEM_READ_SECONDS} s.") from None
|
||||
except (OSError, ValueError, KeyError, WebSocketException) as exc:
|
||||
raise Unreachable(f"Couldn't read Home Assistant's system report: {exc}") from None
|
||||
|
||||
@@ -5,23 +5,35 @@ update as the page's Install does. The sign-in Steward holds is never returned.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.core import weekly
|
||||
from steward.core.formatting import zone
|
||||
from steward.mcp._context import current_caller, mcp_app
|
||||
from steward.mcp.tools._audit import audit_mcp, mcp_username
|
||||
from steward.mcp.tools._shape import age_s, db_session, iso, now_utc, redact_url
|
||||
from steward.mcp.tools.overview import finding
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from . import metrics
|
||||
from .models import HomeAssistantConnection, HomeAssistantUpdate
|
||||
from . import schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .routes import connection_says
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
|
||||
from .updates import CannotStart, start_update
|
||||
|
||||
RECENT_UPDATES = 10
|
||||
|
||||
|
||||
def _schedule_view(plan, now) -> dict | None:
|
||||
if plan is None:
|
||||
return None
|
||||
return {"enabled": plan.enabled, "when": weekly.describe(plan),
|
||||
"next_slot_at": iso(weekly.next_slot(plan, now)) if plan.enabled else None,
|
||||
"last_slot_at": iso(weekly.fired_slot(plan)), "stopped": plan.stopped}
|
||||
|
||||
|
||||
def _run_view(r: HomeAssistantUpdate) -> dict:
|
||||
return {"update_id": r.id, "entity_id": r.entity_id, "title": r.title, "kind": r.kind,
|
||||
"from_version": r.from_version, "to_version": r.to_version, "state": r.state, "message": r.message,
|
||||
@@ -36,16 +48,17 @@ async def get_home_assistant() -> dict:
|
||||
`installable` is what update_home_assistant takes (Core, OS, add-ons);
|
||||
`others_waiting` is firmware and custom integrations, which Steward leaves
|
||||
to Home Assistant. `can_install` is whether the Home Assistant user who signed
|
||||
Steward in is an administrator, which installing needs. `metrics` is the
|
||||
Prometheus target that reads its metrics, `uses_sign_in` saying whether that
|
||||
same user does; null when the Prometheus plugin is off.
|
||||
Steward in is an administrator, which installing needs. `system.problems` is
|
||||
what's wrong with the system itself: open repairs, Supervisor health, low disk,
|
||||
recovery mode, add-ons that should be running. `schedule` is the weekly slot
|
||||
for installing what's waiting, null until one is set.
|
||||
"""
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
|
||||
scrape = await metrics.state(db, c) if c else None
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
if c is None:
|
||||
return {"connected": False, "says": "No connection yet: an admin signs in on the Home Assistant page."}
|
||||
reach, says = connection_says(c, now)
|
||||
@@ -59,7 +72,8 @@ async def get_home_assistant() -> dict:
|
||||
"backup": {"verdict": backup_verdict(backup, now)[1], "last_ok": iso(parse_moment(backup.get("last_ok"))),
|
||||
"age_hours": round(age, 1) if age is not None else None, "stale_after_hours": STALE_BACKUP_HOURS,
|
||||
"next": iso(parse_moment(backup.get("next"))), "last_result": backup.get("last_result")},
|
||||
"metrics": scrape, "updates": [_run_view(r) for r in runs]}
|
||||
"system": c.health or {}, "schedule": _schedule_view(plan, now),
|
||||
"updates": [_run_view(r) for r in runs]}
|
||||
|
||||
|
||||
async def update_home_assistant(entity_id: str) -> dict:
|
||||
@@ -82,9 +96,35 @@ async def update_home_assistant(entity_id: str) -> dict:
|
||||
return {**_run_view(run), "next": "get_home_assistant follows it: `updates` shows its state."}
|
||||
|
||||
|
||||
def overview_findings(c, latest_run, now) -> list[dict]:
|
||||
"""A connection that isn't answering, stale backups, and a newest update that
|
||||
failed. Updates waiting aren't a problem to raise."""
|
||||
async def set_home_assistant_schedule(enabled: bool, weekday: int = 5, at_time: str = "05:00",
|
||||
time_zone: str = "UTC") -> dict:
|
||||
"""Set the weekly slot for installing what's waiting, or turn it off, as the page does.
|
||||
|
||||
weekday: 0 for Monday to 6 for Sunday. at_time: "HH:MM" in time_zone.
|
||||
|
||||
At each slot Steward installs add-ons, then Core, then the OS, one at a time,
|
||||
each after a confirmed backup, stopping at the first failure. A Core update
|
||||
restarts Home Assistant and an OS update reboots the machine, with nobody
|
||||
watching. The first slot is the next one from now. Admin only, and audited.
|
||||
"""
|
||||
try:
|
||||
weekly.parse_at(at_time)
|
||||
except ValueError:
|
||||
raise ValueError(f"at_time is HH:MM, not {at_time!r}.") from None
|
||||
if not 0 <= weekday <= 6:
|
||||
raise ValueError("weekday is 0 for Monday to 6 for Sunday.")
|
||||
if zone(time_zone) is timezone.utc and time_zone.upper() != "UTC":
|
||||
raise ValueError(f"Unknown time zone {time_zone!r}.")
|
||||
row = await schedule.save(mcp_app(), enabled=enabled, weekday=weekday, at_time=at_time, time_zone=time_zone)
|
||||
await audit_mcp("homeassistant.schedule_saved", "homeassistant_schedule", row.id,
|
||||
{"enabled": row.enabled, "when": weekly.describe(row)})
|
||||
return _schedule_view(row, now_utc())
|
||||
|
||||
|
||||
def overview_findings(c, latest_run, now, plan=None) -> list[dict]:
|
||||
"""A connection that isn't answering, a problem with the system, stale backups,
|
||||
a newest update that failed, and a scheduled run that stopped. Updates waiting
|
||||
aren't a problem to raise."""
|
||||
if c is None:
|
||||
return []
|
||||
found = []
|
||||
@@ -92,6 +132,11 @@ def overview_findings(c, latest_run, now) -> list[dict]:
|
||||
if reach == "down":
|
||||
found.append(finding("warning", "home_assistant_down", "Home Assistant",
|
||||
f"Home Assistant isn't answering: {says}", since=c.last_ok_at, next="get_home_assistant()"))
|
||||
problems = problems_open(c.health)
|
||||
if problems:
|
||||
more = f" (and {len(problems) - 1} more)" if len(problems) > 1 else ""
|
||||
found.append(finding("warning", "home_assistant_problem", "Home Assistant", f"{problems[0]['title']}{more}",
|
||||
next="get_home_assistant()"))
|
||||
tone, backup_says = backup_verdict(c.backup, now)
|
||||
if c.last_ok_at is not None and tone == "warn":
|
||||
found.append(finding("warning", "home_assistant_backup", "Home Assistant", backup_says,
|
||||
@@ -100,6 +145,10 @@ def overview_findings(c, latest_run, now) -> list[dict]:
|
||||
found.append(finding("warning", "home_assistant_update_failed", latest_run.title,
|
||||
f"The update of {latest_run.title} failed: {latest_run.message}",
|
||||
since=latest_run.finished_at, next="get_home_assistant()"))
|
||||
if plan is not None and plan.stopped:
|
||||
found.append(finding("warning", "home_assistant_schedule_stopped", "Home Assistant",
|
||||
f"The scheduled update run stopped: {plan.stopped}", since=weekly.fired_slot(plan),
|
||||
next="get_home_assistant()"))
|
||||
return found
|
||||
|
||||
|
||||
@@ -107,9 +156,11 @@ async def mcp_overview(db) -> list[dict]:
|
||||
c = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
latest = (await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
|
||||
return overview_findings(c, latest, now_utc())
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
return overview_findings(c, latest, now_utc(), plan)
|
||||
|
||||
|
||||
def register_tools(register) -> None:
|
||||
register(get_home_assistant, role=UserRole.viewer, read_only=True)
|
||||
register(update_home_assistant, role=UserRole.admin, read_only=False)
|
||||
register(set_home_assistant_schedule, role=UserRole.admin, read_only=False)
|
||||
|
||||
@@ -1,153 +0,0 @@
|
||||
# plugins/homeassistant/metrics.py
|
||||
"""Home Assistant's metrics, read with the one sign-in Steward already holds, so a
|
||||
single Home Assistant user covers everything. The scrape stays a Prometheus target:
|
||||
this lends it a token at each scrape, sends that token only to this Home Assistant,
|
||||
and stops it when the sign-in has been refused."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import or_, select, update
|
||||
|
||||
from steward.core.plugin_manager import loaded_plugin_submodule
|
||||
from steward.core.prometheus import TokenUnavailable
|
||||
|
||||
from . import client
|
||||
from .models import HomeAssistantConnection
|
||||
from .service import SOURCE
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from quart import Quart
|
||||
|
||||
METRICS_PATH = "/api/prometheus"
|
||||
NAME = "HomeAssistant"
|
||||
INTERVAL_SECONDS = 120
|
||||
TRY_SECONDS = 30
|
||||
# What a new target leaves out: bookkeeping on every entity, a few thousand series
|
||||
# that say nothing the readings don't.
|
||||
DROP = ["homeassistant_state_change*", "homeassistant_last_updated_time_seconds", "homeassistant_entity_available",
|
||||
"homeassistant_entity_info", "homeassistant_area_info", "python_*", "process_*"]
|
||||
REFUSED = "Home Assistant refused Steward's sign-in while reading its metrics, or has banned this address: look in its ip_bans.yaml."
|
||||
|
||||
|
||||
class CannotRead(Exception):
|
||||
"""Why the metrics can't be read with this sign-in, said to the person."""
|
||||
|
||||
|
||||
def metrics_url(connection) -> str:
|
||||
return connection.url.rstrip("/") + METRICS_PATH
|
||||
|
||||
|
||||
# ── Lending the sign-in to a scrape ──────────────────────────────────────────
|
||||
|
||||
async def scrape_token(app: "Quart", url: str) -> str:
|
||||
"""A token for scraping `url`. Raises TokenUnavailable, asking nothing of
|
||||
anyone, once the sign-in was refused: another refused request counts toward a
|
||||
ban. A target edited to point elsewhere isn't sent this user's token."""
|
||||
async with app.db_sessionmaker() as db:
|
||||
connection = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is None:
|
||||
raise TokenUnavailable("Home Assistant isn't connected: sign in on its page to read its metrics.")
|
||||
if connection.refused:
|
||||
raise TokenUnavailable(f"{connection.last_error} Steward has stopped asking until you sign in again "
|
||||
"or choose Try again on the Home Assistant page.")
|
||||
if url != metrics_url(connection):
|
||||
raise TokenUnavailable(f"This isn't the address of the Home Assistant Steward signed in to, so its sign-in "
|
||||
f"isn't sent here. Set the address to {metrics_url(connection)}, or type a token.")
|
||||
try:
|
||||
return await client.access_token(connection)
|
||||
except (client.Refused, client.Unreachable) as exc:
|
||||
raise TokenUnavailable(str(exc)) from None
|
||||
|
||||
|
||||
async def token_refused(app: "Quart") -> None:
|
||||
"""A scrape's token was refused: stop everything using this sign-in."""
|
||||
client.forget_access()
|
||||
async with app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
connection = (await db.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is not None:
|
||||
connection.refused, connection.last_error = True, REFUSED
|
||||
|
||||
|
||||
# ── The Metrics section of the page ──────────────────────────────────────────
|
||||
|
||||
def _prometheus(sub: str):
|
||||
return loaded_plugin_submodule("prometheus", sub)
|
||||
|
||||
|
||||
async def _target(db, connection):
|
||||
"""The target using this sign-in, or else the one at this address."""
|
||||
T = _prometheus("models").PrometheusTarget
|
||||
rows = (await db.execute(select(T).where(or_(T.token_from == SOURCE, T.url == metrics_url(connection))))).scalars().all()
|
||||
return next((t for t in rows if t.token_from == SOURCE), rows[0] if rows else None)
|
||||
|
||||
|
||||
async def state(db, connection) -> dict | None:
|
||||
"""What the page says about the metrics. None when the Prometheus plugin is off;
|
||||
`id` is None when no target reads them yet."""
|
||||
service = _prometheus("service")
|
||||
if service is None or _prometheus("models") is None:
|
||||
return None
|
||||
target = await _target(db, connection)
|
||||
if target is None:
|
||||
return {"id": None, "name": None, "uses_sign_in": False, "reach": None, "says": None, "kept": 0}
|
||||
found = service.target_state(target, datetime.now(timezone.utc))
|
||||
return {"id": target.id, "name": target.name, "uses_sign_in": target.token_from == SOURCE,
|
||||
"reach": found.reach, "says": found.says, "kept": target.last_kept}
|
||||
|
||||
|
||||
async def try_reading(app: "Quart", connection) -> None:
|
||||
"""Ask for the metrics once, so turning this on says at once if they aren't there."""
|
||||
scraper = _prometheus("scraper")
|
||||
if scraper is None:
|
||||
raise CannotRead("the Prometheus plugin isn't turned on.")
|
||||
if connection.refused:
|
||||
raise CannotRead("Home Assistant has refused this sign-in. Choose Try again or sign in again first.")
|
||||
try:
|
||||
token = await client.access_token(connection)
|
||||
await asyncio.wait_for(scraper.scrape(metrics_url(connection), token, connection.verify_tls), TRY_SECONDS)
|
||||
except scraper.NoMetricsHere:
|
||||
raise CannotRead("Home Assistant isn't serving metrics. Add prometheus: to its configuration.yaml, restart it, "
|
||||
"and come back: Steward doesn't change Home Assistant.") from None
|
||||
except scraper.TokenRefused:
|
||||
await token_refused(app)
|
||||
raise CannotRead(REFUSED) from None
|
||||
except TimeoutError:
|
||||
raise CannotRead(f"Home Assistant's metrics didn't arrive within {TRY_SECONDS} s.") from None
|
||||
except (scraper.ScrapeError, client.Refused, client.Unreachable) as exc:
|
||||
raise CannotRead(str(exc)) from None
|
||||
|
||||
|
||||
async def use_sign_in(app: "Quart", connection) -> tuple[str, str, bool]:
|
||||
"""Make the target, or move the one at this address onto this sign-in, keeping
|
||||
its name, filters and history. Returns (id, name, whether it was made)."""
|
||||
models, scheduler = _prometheus("models"), _prometheus("scheduler")
|
||||
T = models.PrometheusTarget
|
||||
async with app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
target = await _target(db, connection)
|
||||
created = target is None
|
||||
if created:
|
||||
if (await db.execute(select(T.id).where(T.name == NAME))).first():
|
||||
raise CannotRead(f"another Prometheus target is already named {NAME}. Rename it first.")
|
||||
target = T(name=NAME, interval_seconds=INTERVAL_SECONDS, drop=list(DROP))
|
||||
db.add(target)
|
||||
target.url, target.verify_tls = metrics_url(connection), connection.verify_tls
|
||||
target.token, target.token_from, target.refused = "", SOURCE, False
|
||||
target.last_scraped_at = target.last_error = None
|
||||
await db.flush()
|
||||
target_id, name = target.id, target.name
|
||||
scheduler.RATES.forget(target_id)
|
||||
return target_id, name, created
|
||||
|
||||
|
||||
async def follow_sign_in(db, connection) -> None:
|
||||
"""Signing in at a new address takes the scrape using this sign-in with it."""
|
||||
models = _prometheus("models")
|
||||
if models is not None:
|
||||
T = models.PrometheusTarget
|
||||
await db.execute(update(T).where(T.token_from == SOURCE).values(
|
||||
url=metrics_url(connection), verify_tls=connection.verify_tls))
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Home Assistant: what the poll reads of the system, and a weekly slot for updates
|
||||
|
||||
Revision ID: homeassistant_003_health
|
||||
Revises: homeassistant_002_sign_in
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "homeassistant_003_health"
|
||||
down_revision: Union[str, None] = "homeassistant_002_sign_in"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("homeassistant_connection",
|
||||
sa.Column("health", sa.JSON, nullable=False, server_default="{}"))
|
||||
op.create_table(
|
||||
"homeassistant_schedule",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("enabled", sa.Boolean, nullable=False),
|
||||
sa.Column("weekday", sa.Integer, nullable=False),
|
||||
sa.Column("at_time", sa.String(5), nullable=False),
|
||||
sa.Column("time_zone", sa.String(64), nullable=False),
|
||||
sa.Column("window_minutes", sa.Integer, nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("last_slot_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("stopped", sa.Text, nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("homeassistant_schedule")
|
||||
op.drop_column("homeassistant_connection", "health")
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Index, String, Text, text
|
||||
from sqlalchemy import JSON, Boolean, DateTime, Index, Integer, String, Text, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from steward.core.ids import time_ordered_id
|
||||
@@ -17,7 +17,7 @@ class HomeAssistantConnection(Base):
|
||||
has the same access as the user who signed in, whatever that is: Home
|
||||
Assistant has no narrower permission. `can_install` is whether that user was
|
||||
an administrator, which backups need. The last_* columns and `updates` and
|
||||
`backup` are the newest poll's outcome, which the pages read. `refused` stops
|
||||
`backup` and `health` are the newest poll's outcome, which the pages read. `refused` stops
|
||||
all polling until someone signs in again or tries again: Home Assistant bans
|
||||
an address after a few refused requests.
|
||||
"""
|
||||
@@ -37,6 +37,25 @@ class HomeAssistantConnection(Base):
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
updates: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
backup: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
health: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
|
||||
|
||||
class HomeAssistantSchedule(Base):
|
||||
"""The weekly slot for installing what's waiting; at most one row, none until an
|
||||
admin sets it. `last_slot_at` is the slot that last fired, and `stopped` says why
|
||||
that slot ended early. Times are read through steward.core.weekly."""
|
||||
__tablename__ = "homeassistant_schedule"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
weekday: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
at_time: Mapped[str] = mapped_column(String(5), nullable=False)
|
||||
time_zone: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
window_minutes: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))
|
||||
last_slot_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
stopped: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
|
||||
class HomeAssistantUpdate(Base):
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# plugins/homeassistant/plugin.yaml
|
||||
name: homeassistant
|
||||
version: "1.0.0"
|
||||
description: "Shows which Home Assistant updates are waiting and how old its last backup is, and installs Core, OS and add-on updates after a confirmed backup."
|
||||
description: "Keeps Home Assistant running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates after a confirmed backup, on a click or a weekly schedule."
|
||||
author: "Steward"
|
||||
license: "MIT"
|
||||
min_app_version: "0.1.0"
|
||||
|
||||
@@ -13,14 +13,16 @@ from quart import Blueprint, current_app, redirect, render_template, request, se
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.core import weekly
|
||||
from steward.core.audit import log_audit
|
||||
from steward.core.crypto import encrypt_secret
|
||||
from steward.core.formatting import zone
|
||||
from steward.core.settings import public_base_url
|
||||
from steward.models.users import UserRole
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
from . import client, metrics
|
||||
from .models import HomeAssistantConnection, HomeAssistantUpdate
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment
|
||||
from . import client, schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open
|
||||
from .updates import CannotStart, start_update
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -55,23 +57,29 @@ async def _render_index(error: str | None = None, status: int = 200):
|
||||
connection = await _connection(db)
|
||||
runs = list((await db.execute(select(HomeAssistantUpdate).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars())
|
||||
scrape = await metrics.state(db, connection) if connection else None
|
||||
plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
me = await db.get(User, session.get("user_id")) if session.get("user_id") else None
|
||||
if connection is None:
|
||||
return await render_template("homeassistant/index.html", connection=None, error=error), status
|
||||
default_zone = (me.time_zone if me and me.time_zone else "UTC")
|
||||
now = datetime.now(timezone.utc)
|
||||
reach, says = connection_says(connection, now)
|
||||
updates = connection.updates or []
|
||||
backup = connection.backup or {}
|
||||
tone, backup_says = backup_verdict(backup, now)
|
||||
return await render_template(
|
||||
"homeassistant/index.html", connection=connection, error=error, reach=reach, says=says, metrics=scrape,
|
||||
"homeassistant/index.html", connection=connection, error=error, reach=reach, says=says,
|
||||
installable=installable(connection),
|
||||
others=[u for u in updates if u["waiting"] and not u["kind"]],
|
||||
current=[u for u in updates if not u["waiting"] and u["kind"]],
|
||||
runs=runs, under_way=next((r for r in runs if r.state in ("backing_up", "installing")), None),
|
||||
backup=backup, backup_tone=tone, backup_says=backup_says,
|
||||
backup_at=parse_moment(backup.get("last_ok")), backup_next=parse_moment(backup.get("next")),
|
||||
backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS), status
|
||||
backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS,
|
||||
health=connection.health or {}, problems=problems_open(connection.health),
|
||||
plan=plan, plan_when=weekly.describe(plan) if plan else None,
|
||||
plan_next=weekly.next_slot(plan, now) if plan and plan.enabled else None,
|
||||
weekdays=weekly.WEEKDAYS, default_zone=default_zone), status
|
||||
|
||||
|
||||
@homeassistant_bp.get("/")
|
||||
@@ -96,23 +104,23 @@ async def install():
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
@homeassistant_bp.post("/metrics")
|
||||
@homeassistant_bp.post("/schedule")
|
||||
@require_role(UserRole.admin)
|
||||
async def connect_metrics():
|
||||
"""Read Home Assistant's metrics with this sign-in, after asking for them once
|
||||
to show they're there. An admin's explicit action, so it may ask."""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
connection = await _connection(db)
|
||||
if connection is None:
|
||||
return redirect("/plugins/homeassistant/")
|
||||
async def save_schedule():
|
||||
"""Set the weekly slot for installing what's waiting, or turn it off."""
|
||||
form = await request.form
|
||||
time_zone = (form.get("time_zone") or "UTC").strip()
|
||||
try:
|
||||
await metrics.try_reading(current_app, connection)
|
||||
target_id, name, created = await metrics.use_sign_in(current_app, connection)
|
||||
except metrics.CannotRead as exc:
|
||||
return await _render_index(f"Nothing changed: {exc}", 409)
|
||||
await log_audit(current_app, *_actor(), "prometheus.target_created" if created else "prometheus.target_updated",
|
||||
entity_type="prometheus_target", entity_id=target_id,
|
||||
detail={"name": name, "token_from": metrics.SOURCE})
|
||||
weekday, at_time = int(form.get("weekday", "")), (form.get("at_time") or "").strip()
|
||||
weekly.parse_at(at_time)
|
||||
except ValueError:
|
||||
return await _render_index("Choose a day and a time.", 422)
|
||||
if not 0 <= weekday <= 6 or (zone(time_zone) is timezone.utc and time_zone.upper() != "UTC"):
|
||||
return await _render_index(f"Choose a day, and a known time zone rather than “{time_zone}”.", 422)
|
||||
row = await schedule.save(current_app, enabled=bool(form.get("enabled")), weekday=weekday,
|
||||
at_time=at_time, time_zone=time_zone)
|
||||
await log_audit(current_app, *_actor(), "homeassistant.schedule_saved", entity_type="homeassistant_schedule",
|
||||
entity_id=row.id, detail={"enabled": row.enabled, "when": weekly.describe(row)})
|
||||
return redirect("/plugins/homeassistant/")
|
||||
|
||||
|
||||
@@ -188,7 +196,6 @@ async def sign_in_callback():
|
||||
connection.can_install, connection.signed_in_by = can_install, username
|
||||
connection.signed_in_at = datetime.now(timezone.utc)
|
||||
connection.refused, connection.last_error, connection.last_polled_at = False, None, None
|
||||
await metrics.follow_sign_in(db, connection)
|
||||
await db.flush()
|
||||
connection_id = connection.id
|
||||
if previous is not None:
|
||||
@@ -214,9 +221,7 @@ async def _end_sign_in(connection) -> bool:
|
||||
@homeassistant_bp.post("/connection/retry")
|
||||
@require_role(UserRole.admin)
|
||||
async def retry_connection():
|
||||
"""Ask again after a refusal, once a ban has been dealt with. A metrics scrape
|
||||
using this sign-in starts again with it."""
|
||||
client.forget_access()
|
||||
"""Ask again after a refusal, once a ban has been dealt with."""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
connection = await _connection(db)
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# plugins/homeassistant/schedule.py
|
||||
"""Installing what's waiting at a weekly slot: one at a time through the same path
|
||||
as the page's Install, each after its own confirmed backup. Add-ons go first and
|
||||
the OS last, as it reboots the machine. The first failure ends the slot, leaving the
|
||||
rest for someone to look at. Everything is on the rows, so a restart of Steward or
|
||||
of Home Assistant only delays the next step."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.core import weekly
|
||||
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import installable
|
||||
from .updates import CannotStart, start_update, under_way
|
||||
|
||||
ACTOR = "schedule"
|
||||
WINDOW_MINUTES = 240
|
||||
_ORDER = {"addon": 0, "core": 1, "os": 2}
|
||||
|
||||
|
||||
def next_to_install(waiting: list[dict], tried: set[str]) -> dict | None:
|
||||
left = [u for u in waiting if u["entity_id"] not in tried and not u["in_progress"]]
|
||||
return min(left, key=lambda u: _ORDER[u["kind"]], default=None)
|
||||
|
||||
|
||||
async def _stop(app, schedule_id: str, why: str) -> None:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantSchedule, schedule_id)
|
||||
if row is not None:
|
||||
row.stopped = why
|
||||
|
||||
|
||||
async def _fire(app, schedule_id: str, slot: datetime) -> None:
|
||||
"""Open the slot, and have the next poll say what's waiting at this moment."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = await session.get(HomeAssistantSchedule, schedule_id)
|
||||
row.last_slot_at, row.stopped = slot, None
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is not None:
|
||||
connection.last_polled_at = None
|
||||
|
||||
|
||||
async def step(app, transport=None) -> None:
|
||||
"""Fire a slot that has come, or start the next update of one that's open."""
|
||||
now = datetime.now(timezone.utc)
|
||||
async with app.db_sessionmaker() as session:
|
||||
schedule = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if schedule is None or connection is None:
|
||||
return
|
||||
slot = weekly.is_due(schedule, now)
|
||||
if slot is not None:
|
||||
await _fire(app, schedule.id, slot)
|
||||
return
|
||||
slot = weekly.fired_slot(schedule)
|
||||
if not schedule.enabled or schedule.stopped or slot is None \
|
||||
or now - slot >= timedelta(minutes=schedule.window_minutes):
|
||||
return
|
||||
# Only on a fresh, good read: Home Assistant may be restarting from the last update.
|
||||
if connection.refused or connection.last_polled_at is None or connection.last_error:
|
||||
return
|
||||
if await under_way(session) is not None:
|
||||
return
|
||||
runs = (await session.execute(select(HomeAssistantUpdate).where(
|
||||
HomeAssistantUpdate.started_at >= slot))).scalars().all()
|
||||
failed = next((r for r in runs if r.state == "failed"), None)
|
||||
if failed is not None:
|
||||
await _stop(app, schedule.id, f"{failed.title}: {failed.message} Nothing more was installed in this slot.")
|
||||
return
|
||||
update = next_to_install(installable(connection), {r.entity_id for r in runs})
|
||||
if update is None:
|
||||
return
|
||||
try:
|
||||
await start_update(app, update["entity_id"], ACTOR, transport)
|
||||
except CannotStart as exc:
|
||||
await _stop(app, schedule.id, f"{update['title']} wasn't started: {exc}. Nothing more was installed in this slot.")
|
||||
|
||||
|
||||
async def save(app, *, enabled: bool, weekday: int, at_time: str, time_zone: str) -> HomeAssistantSchedule:
|
||||
"""Set the slot. Its first is the next one from now, not one already past."""
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
row = (await session.execute(select(HomeAssistantSchedule))).scalars().first()
|
||||
if row is None:
|
||||
row = HomeAssistantSchedule(window_minutes=WINDOW_MINUTES)
|
||||
session.add(row)
|
||||
row.enabled, row.weekday, row.time_zone = enabled, weekday, time_zone
|
||||
row.at_time = f"{weekly.parse_at(at_time):%H:%M}"
|
||||
row.last_slot_at, row.stopped = datetime.now(timezone.utc), None
|
||||
return row
|
||||
@@ -10,9 +10,9 @@ from sqlalchemy import select
|
||||
|
||||
from steward.core.scheduler import ScheduledTask
|
||||
|
||||
from . import client
|
||||
from .models import HomeAssistantConnection
|
||||
from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_snapshot
|
||||
from . import client, schedule
|
||||
from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate
|
||||
from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_health, read_snapshot
|
||||
from .updates import advance
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -36,7 +36,7 @@ def make_tick_task(app: "Quart") -> ScheduledTask:
|
||||
if busy.locked():
|
||||
return
|
||||
async with busy:
|
||||
for step in (poll, advance):
|
||||
for step in (poll, advance, schedule.step):
|
||||
try:
|
||||
await asyncio.wait_for(step(app), STEP_DEADLINE_SECONDS)
|
||||
except TimeoutError:
|
||||
@@ -56,7 +56,15 @@ def is_due(connection, now: datetime) -> bool:
|
||||
return (now - connection.last_polled_at).total_seconds() >= POLL_SECONDS - TICK_SECONDS / 2
|
||||
|
||||
|
||||
async def poll(app: "Quart", transport=None) -> None:
|
||||
async def update_failed(session) -> bool:
|
||||
"""Whether the newest update failed, or the schedule couldn't start one."""
|
||||
stopped = (await session.execute(select(HomeAssistantSchedule.stopped))).scalars().first()
|
||||
newest = (await session.execute(select(HomeAssistantUpdate.state).order_by(
|
||||
HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first()
|
||||
return bool(stopped) or newest == "failed"
|
||||
|
||||
|
||||
async def poll(app: "Quart", transport=None, connect=None) -> None:
|
||||
from steward.core.alerts import MetricBatch
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
@@ -64,12 +72,17 @@ async def poll(app: "Quart", transport=None) -> None:
|
||||
connection = (await session.execute(select(HomeAssistantConnection))).scalars().first()
|
||||
if connection is None or not is_due(connection, now):
|
||||
return
|
||||
snapshot, error, refused = None, None, False
|
||||
snapshot, health, error, refused = None, None, None, False
|
||||
try:
|
||||
token = await client.access_token(connection, transport)
|
||||
snapshot = read_snapshot(await client.read_states(connection.url, token, connection.verify_tls, transport))
|
||||
try:
|
||||
health = read_health(await client.read_system(connection.url, token, connection.verify_tls, connect))
|
||||
except client.Unreachable as exc:
|
||||
# Updates and backups were read, so it's answering. The last report stands, and says it's old.
|
||||
health = {**(connection.health or {}), "error": str(exc)}
|
||||
except client.Refused as exc:
|
||||
error, refused = str(exc), True
|
||||
snapshot, error, refused = None, str(exc), True
|
||||
except client.Unreachable as exc:
|
||||
error = str(exc)
|
||||
async with app.db_sessionmaker() as session:
|
||||
@@ -83,7 +96,7 @@ async def poll(app: "Quart", transport=None) -> None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=0.0, recorded_at=now)
|
||||
else:
|
||||
row.last_ok_at = now
|
||||
row.updates, row.backup = snapshot["updates"], snapshot["backup"]
|
||||
row.updates, row.backup, row.health = snapshot["updates"], snapshot["backup"], health
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=1.0, recorded_at=now)
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="updates_waiting",
|
||||
value=len(installable(row)), recorded_at=now)
|
||||
@@ -91,4 +104,13 @@ async def poll(app: "Quart", transport=None) -> None:
|
||||
if age is not None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="backup_age_hours",
|
||||
value=age, recorded_at=now)
|
||||
if not health.get("error"):
|
||||
if health["repairs_read"]:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="problems_open",
|
||||
value=len(health["problems"]), recorded_at=now)
|
||||
if health["disk_free_pct"] is not None:
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="disk_free_pct",
|
||||
value=health["disk_free_pct"], recorded_at=now)
|
||||
batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="update_failed",
|
||||
value=1.0 if await update_failed(session) else 0.0, recorded_at=now)
|
||||
await batch.record(session)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# plugins/homeassistant/service.py
|
||||
"""Reading what Home Assistant reports: which updates it knows of, which of them
|
||||
Steward may install, and how its backups stand."""
|
||||
Steward may install, how its backups stand, and whether the system has a problem."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
@@ -9,6 +9,7 @@ SOURCE = "homeassistant"
|
||||
RESOURCE = "homeassistant"
|
||||
# Backups run daily, so past a day and a bit one has been missed.
|
||||
STALE_BACKUP_HOURS = 26
|
||||
LOW_DISK_PCT = 10
|
||||
|
||||
CORE = "update.home_assistant_core_update"
|
||||
OS = "update.home_assistant_operating_system_update"
|
||||
@@ -108,3 +109,50 @@ def backup_verdict(backup: dict | None, now: datetime) -> tuple[str, str]:
|
||||
if age > STALE_BACKUP_HOURS:
|
||||
return "warn", f"The last good backup is {age:.0f} hours old, past the {STALE_BACKUP_HOURS} hours expected."
|
||||
return "ok", "Backups are up to date."
|
||||
|
||||
|
||||
def _issue_title(issue: dict, titles: dict) -> str:
|
||||
key = issue.get("translation_key") or issue.get("issue_id") or ""
|
||||
title = titles.get(f"component.{issue.get('domain')}.issues.{key}.title")
|
||||
if not title:
|
||||
return key.removeprefix("issue_").replace("_", " ").capitalize() or "A repair is open"
|
||||
for name, value in (issue.get("translation_placeholders") or {}).items():
|
||||
title = title.replace("{" + name + "}", str(value))
|
||||
return title
|
||||
|
||||
|
||||
def read_health(found: dict) -> dict:
|
||||
"""What a poll keeps of client.read_system: the problems to show, and the disk.
|
||||
A part Home Assistant wouldn't answer is left out rather than read as fine:
|
||||
`repairs_read` says whether the repairs were seen, and `supervised` whether a
|
||||
Supervisor answered, as it doesn't on every install or for every user."""
|
||||
problems = []
|
||||
if (found.get("config") or {}).get("recovery_mode"):
|
||||
problems.append({"kind": "recovery", "severity": "critical",
|
||||
"title": "Home Assistant started in recovery mode: its configuration didn't load."})
|
||||
issues = found.get("issues")
|
||||
for issue in issues or []:
|
||||
if not issue.get("ignored") and not issue.get("dismissed_version"):
|
||||
problems.append({"kind": "repair", "severity": issue.get("severity") or "warning",
|
||||
"title": _issue_title(issue, found.get("titles") or {})})
|
||||
resolution = found.get("resolution") or {}
|
||||
for word in ("unhealthy", "unsupported"):
|
||||
for reason in resolution.get(word) or []:
|
||||
problems.append({"kind": "supervisor", "severity": "error" if word == "unhealthy" else "warning",
|
||||
"title": f"The Supervisor calls this system {word}: {str(reason).replace('_', ' ')}."})
|
||||
host = found.get("host") or {}
|
||||
free, total = host.get("disk_free"), host.get("disk_total")
|
||||
free_pct = round(100 * free / total, 1) if free is not None and total else None
|
||||
if free_pct is not None and free_pct < LOW_DISK_PCT:
|
||||
problems.append({"kind": "disk", "severity": "error", "title": f"Only {free:g} GB of {total:g} GB free."})
|
||||
boots = found.get("boots") or {}
|
||||
for addon in found.get("addons") or []:
|
||||
if boots.get(addon.get("slug")) == "auto":
|
||||
problems.append({"kind": "addon", "severity": "error",
|
||||
"title": f"{addon.get('name') or addon.get('slug')} should start on its own and isn't running."})
|
||||
return {"repairs_read": issues is not None, "supervised": found.get("resolution") is not None, "problems": problems,
|
||||
"disk_free_gb": free, "disk_total_gb": total, "disk_free_pct": free_pct}
|
||||
|
||||
|
||||
def problems_open(health: dict | None) -> list[dict]:
|
||||
return (health or {}).get("problems") or []
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.9rem;">
|
||||
<input type="checkbox" name="verify_tls" value="on"{% if verify_tls %} checked{% endif %}> Check its HTTPS certificate
|
||||
</label>
|
||||
<p class="tone-warn"><strong>Steward stores a credential with the same access as the user who signs in.</strong> Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status and metrics, take backups and install updates.</p>
|
||||
<p class="tone-warn"><strong>Steward stores a credential with the same access as the user who signs in.</strong> Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates.</p>
|
||||
<p class="field-hint"><strong>We recommend a service account:</strong> a Home Assistant user made just for Steward (Settings → People → Add person). What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.</p>
|
||||
<p class="field-hint" style="margin-bottom:0;">Installing updates needs an administrator. With any other user, Steward shows status only.</p>
|
||||
</section>
|
||||
@@ -38,7 +38,7 @@
|
||||
<section class="card danger-zone" aria-labelledby="ha-remove">
|
||||
<div>
|
||||
<h2 id="ha-remove" class="section-title">Remove this connection</h2>
|
||||
<p class="field-hint">Steward stops reading Home Assistant, its metrics included, forgets its credential and ends the sign-in there.</p>
|
||||
<p class="field-hint">Steward stops reading Home Assistant, forgets its credential and ends the sign-in there.</p>
|
||||
</div>
|
||||
<form method="post" action="/plugins/homeassistant/connection/delete"
|
||||
data-confirm="Remove the Home Assistant connection? Steward stops reading it and forgets its credential.">{{ csrf_field() }}
|
||||
|
||||
@@ -36,12 +36,33 @@
|
||||
{% endif %}
|
||||
|
||||
<div class="stat-strip">
|
||||
{{ stat_tile("System problems", problems|length if health.repairs_read else "—", state="degraded" if problems else none) }}
|
||||
{{ stat_tile("Disk free", (health.disk_free_gb ~ " GB") if health.disk_free_gb is not none and health.disk_free_gb is defined else "—") }}
|
||||
{{ stat_tile("Updates waiting", installable|length if connection.last_ok_at else "—") }}
|
||||
{{ stat_tile("Last good backup", (backup_at|ago) if backup_at else "—", state="degraded" if backup_tone == 'warn' else none) }}
|
||||
{{ stat_tile("Next backup", (backup_next|ago) if backup_next else "—") }}
|
||||
</div>
|
||||
<p class="{{ 'tone-warn' if backup_tone == 'warn' else 'field-hint' }}">{{ backup_says }}</p>
|
||||
|
||||
<section class="card" aria-labelledby="ha-system">
|
||||
<h2 id="ha-system" class="section-title">System</h2>
|
||||
{% if problems %}
|
||||
<table class="table">
|
||||
<thead><tr><th scope="col">Problem</th><th scope="col">Severity</th></tr></thead>
|
||||
<tbody>
|
||||
{% for p in problems %}<tr><td>{{ p.title }}</td><td>{{ status_badge('down' if p.severity in ('critical', 'error') else 'degraded', p.severity|capitalize) }}</td></tr>{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="field-hint">Fix these in Home Assistant: Settings → System → Repairs.</p>
|
||||
{% elif health.repairs_read %}
|
||||
<p class="field-hint">No repairs open, and nothing wrong reported.</p>
|
||||
{% elif connection.last_ok_at %}
|
||||
<p class="tone-warn">Home Assistant wouldn't share its repairs with this user. {% if admin %}<a href="/plugins/homeassistant/connection">Sign in again</a> as an administrator to see them.{% endif %}</p>
|
||||
{% endif %}
|
||||
{% if health.repairs_read and not health.supervised %}<p class="field-hint">No Supervisor answered, so disk space and add-ons aren't checked here.</p>{% endif %}
|
||||
{% if health.error %}<p class="tone-warn" style="margin-bottom:0;">This is from an earlier read. {{ health.error }}</p>{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="card" aria-labelledby="ha-updates">
|
||||
<h2 id="ha-updates" class="section-title">Updates</h2>
|
||||
{% if installable %}
|
||||
@@ -73,21 +94,36 @@
|
||||
{% if others %}<p class="field-hint" style="margin-bottom:0;">Waiting, for Home Assistant to install: {% for u in others %}{{ u.title }}{{ ", " if not loop.last }}{% endfor %}. Steward installs Core, OS and add-on updates only.</p>{% endif %}
|
||||
</section>
|
||||
|
||||
{% if metrics %}
|
||||
<section class="card" aria-labelledby="ha-metrics">
|
||||
<h2 id="ha-metrics" class="section-title">Metrics</h2>
|
||||
{% if metrics.uses_sign_in %}
|
||||
<p class="field-hint" style="margin-bottom:0;">{{ status_badge(metrics.reach) }} Read with this sign-in into the target <a href="/plugins/prometheus/targets/{{ metrics.id }}">{{ metrics.name }}</a>: {{ metrics.says }}{% if metrics.reach == 'up' %} {{ metrics.kept }} series kept.{% endif %}</p>
|
||||
{% elif metrics.id %}
|
||||
<p class="field-hint">The target <a href="/plugins/prometheus/targets/{{ metrics.id }}">{{ metrics.name }}</a> reads these with a token of its own: {{ metrics.says }}</p>
|
||||
{% if admin %}<p class="field-hint">Use this sign-in instead, so one Home Assistant user covers everything. The target keeps its name, filters and history, and forgets its token.</p>
|
||||
<form method="post" action="/plugins/homeassistant/metrics">{{ csrf_field() }}<button class="btn btn-sm" type="submit">Use this sign-in</button></form>{% endif %}
|
||||
<section class="card" aria-labelledby="ha-schedule">
|
||||
<h2 id="ha-schedule" class="section-title">Scheduled updates</h2>
|
||||
{% if plan and plan.enabled %}
|
||||
<p class="field-hint">{{ plan_when }}. Next: {{ plan_next|ago }}. Add-ons first, then Core, then the OS, each after a confirmed backup. A failure ends the run and the rest wait.</p>
|
||||
{% else %}
|
||||
<p class="field-hint">Home Assistant's entity readings can be kept as metrics too, for charts and alert rules, with this sign-in and nothing else to set up.</p>
|
||||
{% if admin %}<form method="post" action="/plugins/homeassistant/metrics">{{ csrf_field() }}<button class="btn btn-sm" type="submit">Read its metrics</button></form>{% endif %}
|
||||
<p class="field-hint">Off. Updates are only installed when someone chooses Install.</p>
|
||||
{% endif %}
|
||||
{% if plan and plan.stopped %}<p class="tone-warn">The last run stopped early. {{ plan.stopped }}</p>{% endif %}
|
||||
{% if admin %}
|
||||
<p class="tone-warn">A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch. An update can also change how automations behave.</p>
|
||||
<form method="post" action="/plugins/homeassistant/schedule" style="display:flex;flex-wrap:wrap;gap:0.75rem;align-items:flex-end;">{{ csrf_field() }}
|
||||
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;margin-bottom:0.4rem;">
|
||||
<input type="checkbox" name="enabled" value="on"{% if plan and plan.enabled %} checked{% endif %}> Install on a schedule
|
||||
</label>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-day">Day</label>
|
||||
<select class="input" id="ha-day" name="weekday">{% for d in weekdays %}<option value="{{ loop.index0 }}"{% if loop.index0 == (plan.weekday if plan else 5) %} selected{% endif %}>{{ d }}</option>{% endfor %}</select>
|
||||
</div>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-time">Time</label>
|
||||
<input class="input" id="ha-time" type="time" name="at_time" required value="{{ plan.at_time if plan else '05:00' }}">
|
||||
</div>
|
||||
<div class="field" style="margin-bottom:0;">
|
||||
<label class="field-label" for="ha-zone">Time zone</label>
|
||||
<input class="input" id="ha-zone" name="time_zone" required value="{{ plan.time_zone if plan else default_zone }}">
|
||||
</div>
|
||||
<button class="btn btn-sm" type="submit">Save</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if runs %}
|
||||
<section class="card" aria-labelledby="ha-runs">
|
||||
|
||||
@@ -21,11 +21,12 @@ the host it runs on, picks up where it was.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, time, timedelta, timezone
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from steward.core.weekly import WEEKDAYS, describe, fired_slot, is_due, last_slot, next_slot, parse_at # noqa: F401
|
||||
|
||||
from . import maintenance, rebalance, timeline
|
||||
|
||||
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
|
||||
# How long a host may take to be back in service after its job finished.
|
||||
SETTLE_LIMIT = timedelta(minutes=30)
|
||||
# An agent heard from this recently is posting.
|
||||
@@ -33,58 +34,6 @@ POSTING_WITHIN = timedelta(minutes=3)
|
||||
RUN_STATES = ("running", "done", "failed", "stopped")
|
||||
|
||||
|
||||
def _zone(name: str):
|
||||
from steward.core.formatting import zone
|
||||
return zone(name)
|
||||
|
||||
|
||||
def parse_at(at_time: str) -> time:
|
||||
h, m = (int(x) for x in at_time.split(":", 1))
|
||||
if not (0 <= h < 24 and 0 <= m < 60):
|
||||
raise ValueError(at_time)
|
||||
return time(h, m)
|
||||
|
||||
|
||||
def last_slot(plan, now: datetime) -> datetime:
|
||||
"""The most recent scheduled time at or before `now`, in UTC."""
|
||||
tz = _zone(plan.time_zone)
|
||||
local = now.astimezone(tz)
|
||||
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
|
||||
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
|
||||
if slot > local:
|
||||
slot -= timedelta(days=7)
|
||||
return slot.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def fired_slot(plan) -> datetime | None:
|
||||
"""The last slot that fired, or None while none has. A new plan's
|
||||
last_slot_at holds the moment it was made, so its first slot is the
|
||||
next one rather than one already past; that moment is no slot (#5863)."""
|
||||
at = plan.last_slot_at
|
||||
if at is None:
|
||||
return None
|
||||
return at if last_slot(plan, at) == at else None
|
||||
|
||||
|
||||
def next_slot(plan, now: datetime) -> datetime:
|
||||
return last_slot(plan, now) + timedelta(days=7)
|
||||
|
||||
|
||||
def is_due(plan, now: datetime) -> datetime | None:
|
||||
"""The slot to fire now, or None. A slot fires once (last_slot_at), and
|
||||
only while its window is still open: a Steward that was down over the slot
|
||||
starts late, not a day late."""
|
||||
if not plan.enabled:
|
||||
return None
|
||||
slot = last_slot(plan, now)
|
||||
since = plan.last_slot_at or plan.created_at
|
||||
if since is not None and slot <= since:
|
||||
return None
|
||||
if now - slot >= timedelta(minutes=plan.window_minutes):
|
||||
return None
|
||||
return slot
|
||||
|
||||
|
||||
def back_in_service(*, last_seen_at, node, services, job_finished_at, now: datetime,
|
||||
expect_active: bool = True) -> str | None:
|
||||
"""None when the host is back in service, else why not yet. `node` is the
|
||||
@@ -134,13 +83,6 @@ def next_move(run, plan, job, back: str | None, now: datetime) -> tuple[str, str
|
||||
return "settling", back
|
||||
|
||||
|
||||
def describe(plan) -> str:
|
||||
""""Sundays at 03:00 (Europe/London), up to 4 h" for the page."""
|
||||
hours = plan.window_minutes / 60
|
||||
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
|
||||
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
|
||||
|
||||
|
||||
# ── applied to the database: the minute tick, Run now, the page ──────────────
|
||||
|
||||
async def group_hosts(session, group_id: str) -> list:
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Every target holds its own token: one that took it from a sign-in goes
|
||||
|
||||
Revision ID: prometheus_005_own_tokens
|
||||
Revises: prometheus_004_token_from
|
||||
Create Date: 2026-11-01
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "prometheus_005_own_tokens"
|
||||
down_revision: Union[str, None] = "prometheus_004_token_from"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Left in place, such a target would ask with no token, and some services ban
|
||||
# an address for that. Its newest values go with it; history ages out.
|
||||
op.execute("""
|
||||
DELETE FROM plugin_metric_latest l USING prometheus_targets t
|
||||
WHERE t.token_from IS NOT NULL AND l.source_module = 'prometheus'
|
||||
AND (l.resource_name = t.name OR left(l.resource_name, length(t.name) + 1) = t.name || ':')""")
|
||||
op.execute("DELETE FROM prometheus_targets WHERE token_from IS NOT NULL")
|
||||
op.drop_column("prometheus_targets", "token_from")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True))
|
||||
@@ -13,8 +13,7 @@ from steward.models.base import Base
|
||||
class PrometheusTarget(Base):
|
||||
"""A /metrics endpoint Steward scrapes, edited on the Prometheus page.
|
||||
|
||||
The bearer token is stored encrypted, or `token_from` names the plugin whose
|
||||
sign-in supplies it at each scrape (its metrics.scrape_token). keep and drop are lists of name
|
||||
The bearer token is stored encrypted. keep and drop are lists of name
|
||||
patterns ("fabledarchives_*") an admin chooses: Steward does the
|
||||
filtering, the endpoint serves everything. The last_* columns are the
|
||||
newest scrape's outcome, written by the scraper, so a page can say why a
|
||||
@@ -27,7 +26,6 @@ class PrometheusTarget(Base):
|
||||
name: Mapped[str] = mapped_column(String(100), nullable=False, unique=True)
|
||||
url: Mapped[str] = mapped_column(String(500), nullable=False)
|
||||
token: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
token_from: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
verify_tls: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
interval_seconds: Mapped[int] = mapped_column(Integer, nullable=False, default=60)
|
||||
keep: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
|
||||
@@ -12,14 +12,13 @@ from sqlalchemy import delete, func, select, update
|
||||
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.core.audit import log_audit
|
||||
from steward.core.crypto import decrypt_secret
|
||||
from steward.core.metric_series import latest_values, points
|
||||
from steward.core.prometheus import TokenUnavailable
|
||||
from steward.models.metrics import PluginMetric, PluginMetricLatest
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from . import library
|
||||
from .models import PrometheusTarget, PrometheusWidgetTemplate
|
||||
from .scheduler import target_token
|
||||
from .scraper import ScrapeError, scrape
|
||||
from .service import (OWN_METRICS, SOURCE, FormError, fold_states, form_values, is_moment, map_families,
|
||||
parse_form, patterns, shown_value, target_state)
|
||||
@@ -81,8 +80,7 @@ def _typed(form, target: PrometheusTarget | None = None) -> dict:
|
||||
return {"name": form.get("name", ""), "url": form.get("url", ""),
|
||||
"interval_seconds": form.get("interval_seconds", ""), "series_cap": form.get("series_cap", ""),
|
||||
"keep": form.get("keep", ""), "drop": form.get("drop", ""),
|
||||
"verify_tls": bool(form.get("verify_tls")), "has_token": bool(target and target.token),
|
||||
"token_from": target.token_from if target else None}
|
||||
"verify_tls": bool(form.get("verify_tls")), "has_token": bool(target and target.token)}
|
||||
|
||||
|
||||
async def _name_taken(db, name: str, other_than: str | None = None) -> bool:
|
||||
@@ -237,15 +235,10 @@ async def test_target():
|
||||
existing = await db.get(PrometheusTarget, form["target_id"])
|
||||
url = (form.get("url") or "").strip()
|
||||
token = (form.get("token") or "").strip()
|
||||
if not token and existing is not None and existing.token and not form.get("clear_token"):
|
||||
token = decrypt_secret(existing.token, context=f"prometheus target {existing.name} token")
|
||||
found: dict = {}
|
||||
if not token and existing is not None and not form.get("clear_token"):
|
||||
try:
|
||||
token = await target_token(current_app, existing, url)
|
||||
except TokenUnavailable as exc:
|
||||
found["error"] = str(exc)
|
||||
if found:
|
||||
pass
|
||||
elif not url.startswith(("http://", "https://")):
|
||||
if not url.startswith(("http://", "https://")):
|
||||
found["error"] = "Enter the address first: http://… or https://…"
|
||||
else:
|
||||
try:
|
||||
|
||||
@@ -48,33 +48,14 @@ def is_due(target, now: datetime) -> bool:
|
||||
return elapsed >= target.interval_seconds - TICK_SECONDS / 2
|
||||
|
||||
|
||||
def _token_source(name: str):
|
||||
from steward.core.plugin_manager import loaded_plugin_submodule
|
||||
return loaded_plugin_submodule(name, "metrics")
|
||||
|
||||
|
||||
async def target_token(app: "Quart", target, url: str) -> str:
|
||||
"""The token to ask `url` with: the target's own, or one from the plugin whose
|
||||
sign-in it uses. Raises TokenUnavailable when that plugin has none to give."""
|
||||
from steward.core.crypto import decrypt_secret
|
||||
from steward.core.prometheus import TokenUnavailable
|
||||
|
||||
if not target.token_from:
|
||||
return decrypt_secret(target.token, context=f"prometheus target {target.name} token")
|
||||
source = _token_source(target.token_from)
|
||||
if source is None:
|
||||
raise TokenUnavailable(f"Its token comes from the {target.token_from} plugin, which isn't turned on.")
|
||||
return await source.scrape_token(app, url)
|
||||
|
||||
|
||||
async def scrape_due(app: "Quart", scrape=None) -> None:
|
||||
"""Scrape every due target at once, then write each one's outcome to its
|
||||
row and its series to the metric tables. `scrape` stands in for
|
||||
scraper.scrape in tests."""
|
||||
from steward.core.prometheus import TokenUnavailable
|
||||
from steward.core.crypto import decrypt_secret
|
||||
|
||||
from .models import PrometheusTarget
|
||||
from .scraper import ScrapeError, TokenRefused
|
||||
from .scraper import ScrapeError
|
||||
from .scraper import scrape as fetch
|
||||
|
||||
scrape = scrape or fetch
|
||||
@@ -87,15 +68,11 @@ async def scrape_due(app: "Quart", scrape=None) -> None:
|
||||
|
||||
async def one(t):
|
||||
started = time.monotonic()
|
||||
|
||||
async def fetched():
|
||||
return await scrape(t.url, await target_token(app, t, t.url), t.verify_tls)
|
||||
|
||||
token = decrypt_secret(t.token, context=f"prometheus target {t.name} token")
|
||||
try:
|
||||
families = await asyncio.wait_for(fetched(), SCRAPE_DEADLINE_SECONDS)
|
||||
families = await asyncio.wait_for(scrape(t.url, token, t.verify_tls),
|
||||
SCRAPE_DEADLINE_SECONDS)
|
||||
return families, (time.monotonic() - started) * 1000
|
||||
except TokenUnavailable as exc:
|
||||
return ScrapeError(str(exc)), (time.monotonic() - started) * 1000
|
||||
except TimeoutError:
|
||||
return ScrapeError(f"Gave up after {SCRAPE_DEADLINE_SECONDS} s."), \
|
||||
(time.monotonic() - started) * 1000
|
||||
@@ -104,11 +81,6 @@ async def scrape_due(app: "Quart", scrape=None) -> None:
|
||||
|
||||
results = await asyncio.gather(*(one(t) for t in targets))
|
||||
await record(app, targets, results, datetime.now(timezone.utc))
|
||||
# A refused sign-in is the lending plugin's to stop, so one Try again or sign-in
|
||||
# there starts everything that uses it.
|
||||
for t, (result, _) in zip(targets, results):
|
||||
if t.token_from and isinstance(result, TokenRefused) and (source := _token_source(t.token_from)):
|
||||
await source.token_refused(app)
|
||||
|
||||
|
||||
async def forget_gone(session, target: str, kept: set[tuple[str, str]]) -> None:
|
||||
@@ -160,7 +132,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None:
|
||||
if not isinstance(result, ScrapeError):
|
||||
logger.warning("Prometheus scrape of %s failed: %r", target.name, result)
|
||||
row.last_error = str(result) or "The scrape failed."
|
||||
row.refused = isinstance(result, TokenRefused) and not row.token_from
|
||||
row.refused = isinstance(result, TokenRefused)
|
||||
batch.add(source_module=SOURCE, resource_name=row.name, metric_name=IS_UP,
|
||||
value=0.0, recorded_at=now)
|
||||
continue
|
||||
|
||||
@@ -23,10 +23,6 @@ class TokenRefused(ScrapeError):
|
||||
Home Assistant, for one, bans an address after a few refusals."""
|
||||
|
||||
|
||||
class NoMetricsHere(ScrapeError):
|
||||
"""The address answered 404: nothing is serving metrics there."""
|
||||
|
||||
|
||||
async def scrape(url: str, token: str = "", verify_tls: bool = True,
|
||||
transport: httpx.AsyncBaseTransport | None = None) -> dict[str, prometheus.Family]:
|
||||
"""The endpoint's families. `transport` stands in for the network in tests."""
|
||||
@@ -40,8 +36,6 @@ async def scrape(url: str, token: str = "", verify_tls: bool = True,
|
||||
if response.status_code in (401, 403):
|
||||
raise TokenRefused(f"Answered {response.status_code}: the token was refused. "
|
||||
"Check the token, or make a new one on the service.")
|
||||
if response.status_code == 404:
|
||||
raise NoMetricsHere(f"Answered 404 {response.reason_phrase}, not metrics.")
|
||||
if response.status_code != 200:
|
||||
raise ScrapeError(f"Answered {response.status_code} {response.reason_phrase}, not metrics.")
|
||||
body = bytearray()
|
||||
|
||||
@@ -212,7 +212,7 @@ def form_values(target: PrometheusTarget | None) -> dict:
|
||||
return {"name": target.name, "url": target.url, "interval_seconds": target.interval_seconds,
|
||||
"series_cap": target.series_cap, "verify_tls": target.verify_tls,
|
||||
"keep": "\n".join(target.keep or []),
|
||||
"drop": "\n".join(target.drop or []), "has_token": bool(target.token), "token_from": target.token_from}
|
||||
"drop": "\n".join(target.drop or []), "has_token": bool(target.token)}
|
||||
|
||||
|
||||
def _bounded(form, key: str, default: int, low: int, high: int, says: str) -> int:
|
||||
@@ -250,7 +250,6 @@ def parse_form(form, existing: PrometheusTarget | None = None) -> dict:
|
||||
else:
|
||||
token = ""
|
||||
return {"name": name, "url": url, "token": token,
|
||||
"token_from": None if typed or existing is None else existing.token_from,
|
||||
"verify_tls": bool(form.get("verify_tls")), "interval_seconds": interval,
|
||||
"keep": patterns(form.get("keep") or ""), "drop": patterns(form.get("drop") or ""),
|
||||
"series_cap": cap}
|
||||
|
||||
@@ -37,12 +37,8 @@
|
||||
<div class="field">
|
||||
<label class="field-label" for="prom-token">Bearer token</label>
|
||||
<input class="input" id="prom-token" name="token" type="password" autocomplete="new-password"
|
||||
placeholder="{{ 'from a sign-in; type to change' if v.token_from else ('kept; type to change' if v.has_token else 'none') }}">
|
||||
{% if v.token_from %}
|
||||
<span class="field-hint">Comes from the sign-in on the {{ v.token_from }} plugin's page, so that one user covers this too. Typing a token here uses that instead.</span>
|
||||
{% else %}
|
||||
placeholder="{{ 'kept; type to change' if v.has_token else 'none' }}">
|
||||
<span class="field-hint">Sent as "Authorization: Bearer …". Blank {{ "keeps the saved one" if v.has_token else "sends none" }}.</span>
|
||||
{% endif %}
|
||||
{% if v.has_token %}
|
||||
<label style="display:flex;gap:0.5rem;align-items:center;margin-top:0.3rem;cursor:pointer;">
|
||||
<input type="checkbox" name="clear_token" value="on"> <span class="field-hint">Remove the saved token</span>
|
||||
|
||||
@@ -17,6 +17,8 @@ dependencies = [
|
||||
"packaging>=24.0",
|
||||
"click>=8.0",
|
||||
"httpx>=0.27",
|
||||
# Floor: websockets.asyncio.client first ships in 13.
|
||||
"websockets>=13",
|
||||
"cryptography>=42",
|
||||
# Floor, not a cap: MCPServer (mcp.server.mcpserver) and the mcp.types
|
||||
# mirror that steward/mcp/ is written against first ship in SDK 2.2.
|
||||
|
||||
@@ -217,6 +217,7 @@ ACTIONS = {
|
||||
"homeassistant.connection_saved": "signed Steward in to Home Assistant",
|
||||
"homeassistant.connection_removed": "removed the Home Assistant connection",
|
||||
"homeassistant.update_started": "started a Home Assistant update",
|
||||
"homeassistant.schedule_saved": "set Home Assistant's update schedule",
|
||||
"syslog.source_added": "added syslog source",
|
||||
"syslog.source_updated": "changed syslog source",
|
||||
"syslog.source_removed": "removed syslog source",
|
||||
|
||||
@@ -30,6 +30,7 @@ KEPT: frozenset[str] = frozenset({
|
||||
"syslog_sources",
|
||||
"syslog_patterns",
|
||||
"homeassistant_connection",
|
||||
"homeassistant_schedule",
|
||||
})
|
||||
|
||||
# Rows that are measurements, logs, history or in-flight work: left out.
|
||||
@@ -94,7 +95,8 @@ ACTIVITY: dict[str, frozenset[str]] = {
|
||||
"prometheus_targets": frozenset({"refused", "last_scraped_at", "last_ok_at", "last_error", "last_kept",
|
||||
"last_dropped", "last_capped"}),
|
||||
"homeassistant_connection": frozenset({"refused", "last_polled_at", "last_ok_at", "last_error",
|
||||
"updates", "backup"}),
|
||||
"updates", "backup", "health"}),
|
||||
"homeassistant_schedule": frozenset({"last_slot_at", "stopped"}),
|
||||
# What the sender last sent; its address, name, state and host are setup.
|
||||
"syslog_sources": frozenset({"hostname_seen", "first_seen_at", "last_seen_at", "lines_total",
|
||||
"last_severity", "last_line"}),
|
||||
@@ -126,5 +128,5 @@ AREAS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("Dashboards", ("dashboards", "dashboard_widgets", "dashboard_share_tokens")),
|
||||
("Plugins", ("snmp_devices", "prometheus_targets", "prometheus_widget_templates",
|
||||
"unifi_known_clients", "syslog_sources", "syslog_patterns",
|
||||
"homeassistant_connection")),
|
||||
"homeassistant_connection", "homeassistant_schedule")),
|
||||
)
|
||||
|
||||
@@ -21,12 +21,6 @@ _LABEL_NAME = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*")
|
||||
|
||||
TYPES = ("counter", "gauge", "histogram", "summary", "untyped")
|
||||
|
||||
|
||||
class TokenUnavailable(Exception):
|
||||
"""A plugin that lends its sign-in to a scrape has no token to give, said to
|
||||
the person. Nothing is sent to the endpoint."""
|
||||
|
||||
|
||||
# Suffixes a histogram or summary's samples carry beyond the family name.
|
||||
_CHILD_SUFFIXES = {
|
||||
"counter": ("_total",), # OpenMetrics declares a counter without it
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# steward/core/weekly.py
|
||||
"""A weekly slot: a weekday and a time in a time zone, with a window after it. A
|
||||
slot fires once, and only while its window is open, so a Steward that was down
|
||||
over the slot starts late rather than a day late. A schedule is anything with
|
||||
weekday, at_time, time_zone, window_minutes, enabled, last_slot_at and created_at."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, time, timedelta, timezone
|
||||
|
||||
from steward.core.formatting import zone
|
||||
|
||||
WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
|
||||
|
||||
|
||||
def parse_at(at_time: str) -> time:
|
||||
h, m = (int(x) for x in at_time.split(":", 1))
|
||||
if not (0 <= h < 24 and 0 <= m < 60):
|
||||
raise ValueError(at_time)
|
||||
return time(h, m)
|
||||
|
||||
|
||||
def last_slot(plan, now: datetime) -> datetime:
|
||||
"""The most recent scheduled time at or before `now`, in UTC."""
|
||||
tz = zone(plan.time_zone)
|
||||
local = now.astimezone(tz)
|
||||
day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7)
|
||||
slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz)
|
||||
if slot > local:
|
||||
slot -= timedelta(days=7)
|
||||
return slot.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def fired_slot(plan) -> datetime | None:
|
||||
"""The last slot that fired, or None while none has. A new schedule's
|
||||
last_slot_at holds the moment it was made, so its first slot is the
|
||||
next one rather than one already past; that moment is no slot."""
|
||||
at = plan.last_slot_at
|
||||
if at is None:
|
||||
return None
|
||||
return at if last_slot(plan, at) == at else None
|
||||
|
||||
|
||||
def next_slot(plan, now: datetime) -> datetime:
|
||||
return last_slot(plan, now) + timedelta(days=7)
|
||||
|
||||
|
||||
def is_due(plan, now: datetime) -> datetime | None:
|
||||
"""The slot to fire now, or None."""
|
||||
if not plan.enabled:
|
||||
return None
|
||||
slot = last_slot(plan, now)
|
||||
since = plan.last_slot_at or plan.created_at
|
||||
if since is not None and slot <= since:
|
||||
return None
|
||||
if now - slot >= timedelta(minutes=plan.window_minutes):
|
||||
return None
|
||||
return slot
|
||||
|
||||
|
||||
def describe(plan) -> str:
|
||||
""""Sundays at 03:00 (Europe/London), up to 4 h" for a page."""
|
||||
hours = plan.window_minutes / 60
|
||||
span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min"
|
||||
return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}"
|
||||
@@ -50,7 +50,6 @@ class FakeHomeAssistant:
|
||||
self.admin = True
|
||||
self.reject_install = None
|
||||
self.cut_off = False
|
||||
self.no_metrics = False
|
||||
self.entities = {
|
||||
ADDON: self._update("on", "1.0"),
|
||||
"sensor.backup_last_successful_automatic_backup": {"state": BEFORE, "attributes": {}},
|
||||
@@ -85,10 +84,6 @@ class FakeHomeAssistant:
|
||||
self.tokens.add(request.headers.get("authorization"))
|
||||
if self.refuse:
|
||||
return httpx.Response(401)
|
||||
if path == "/api/prometheus":
|
||||
if self.no_metrics:
|
||||
return httpx.Response(404)
|
||||
return httpx.Response(200, text="# TYPE homeassistant_entity_x gauge\nhomeassistant_entity_x 1\n")
|
||||
if path == "/api/template":
|
||||
return httpx.Response(200, text="1") if self.admin else httpx.Response(401)
|
||||
if path.startswith("/api/services/"):
|
||||
@@ -119,6 +114,7 @@ async def _clean(app, **connection):
|
||||
async with s.begin():
|
||||
await s.execute(delete(m.HomeAssistantUpdate))
|
||||
await s.execute(delete(m.HomeAssistantConnection))
|
||||
await s.execute(delete(m.HomeAssistantSchedule))
|
||||
s.add(m.HomeAssistantConnection(**{
|
||||
"url": "http://ha.test:8123", "client_id": "http://steward.test/",
|
||||
"refresh_token": encrypt_secret(f"refresh-{uuid.uuid4().hex}"), "can_install": True, **connection}))
|
||||
@@ -471,164 +467,214 @@ def test_signing_in_again_ends_the_sign_in_it_replaces(app, monkeypatch):
|
||||
assert ha.revoked == ["refresh-old"] and not connection.refused and connection.last_polled_at is None
|
||||
|
||||
|
||||
# ── Its metrics, under the one sign-in ───────────────────────────────────────
|
||||
# ── The system, over the websocket ───────────────────────────────────────────
|
||||
|
||||
def _prom(sub):
|
||||
from steward.core.plugin_manager import loaded_plugin_submodule
|
||||
return loaded_plugin_submodule("prometheus", sub)
|
||||
class FakeSocket:
|
||||
"""Home Assistant's websocket: `answers` by question type or Supervisor endpoint,
|
||||
where one left out is refused as it is for a non-administrator."""
|
||||
|
||||
def __init__(self, answers, auth="auth_ok"):
|
||||
self.answers, self.auth = answers, auth
|
||||
self.waiting = [{"type": "auth_required"}]
|
||||
self.asked = []
|
||||
|
||||
def __call__(self, url, **options):
|
||||
self.url = url
|
||||
return self
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc):
|
||||
return False
|
||||
|
||||
async def send(self, text):
|
||||
question = json.loads(text)
|
||||
if question["type"] == "auth":
|
||||
self.waiting.append({"type": self.auth})
|
||||
return
|
||||
key = question.get("endpoint") or question["type"]
|
||||
self.asked.append(key)
|
||||
# Something unasked arriving first must be passed over.
|
||||
self.waiting.append({"type": "event", "id": 9999})
|
||||
self.waiting.append({"id": question["id"], "success": key in self.answers, "result": self.answers.get(key)})
|
||||
|
||||
async def recv(self):
|
||||
return json.dumps(self.waiting.pop(0))
|
||||
|
||||
|
||||
METRICS_URL = "http://ha.test:8123/api/prometheus"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def signed_in(app, monkeypatch):
|
||||
"""A fake Home Assistant behind both the plugin's client and the scraper."""
|
||||
client, scraper = _ha("client"), _prom("scraper")
|
||||
def test_a_poll_keeps_the_system_problems_and_records_them(app):
|
||||
from steward.models.metrics import PluginMetricLatest
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
real_token, real_scrape = client.access_token, scraper.scrape
|
||||
monkeypatch.setattr(client, "access_token", lambda c: real_token(c, ha.transport))
|
||||
monkeypatch.setattr(scraper, "scrape", lambda url, token, verify: real_scrape(url, token, verify, ha.transport))
|
||||
return ha
|
||||
socket = FakeSocket({
|
||||
"get_config": {"recovery_mode": False},
|
||||
"repairs/list_issues": {"issues": [{"domain": "hassio", "translation_key": "issue_x", "severity": "warning"}]},
|
||||
"frontend/get_translations": {"resources": {"component.hassio.issues.issue_x.title": "Something's wrong"}},
|
||||
"/resolution/info": {"unhealthy": [], "unsupported": []},
|
||||
"/host/info": {"disk_free": 50.0, "disk_total": 100.0},
|
||||
"/addons": {"addons": [{"slug": "a", "state": "started"}, {"slug": "b", "name": "B", "state": "stopped"}]},
|
||||
"/addons/b/info": {"boot": "auto"},
|
||||
})
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=socket)
|
||||
connection, _ = await _state(app)
|
||||
async with app.db_sessionmaker() as s:
|
||||
rows = (await s.execute(select(PluginMetricLatest).where(
|
||||
PluginMetricLatest.source_module == "homeassistant"))).scalars().all()
|
||||
return connection, {r.metric_name: r.value for r in rows}
|
||||
|
||||
connection, metrics = asyncio.run(go())
|
||||
assert socket.url == "ws://ha.test:8123/api/websocket"
|
||||
assert "/addons/a/info" not in socket.asked # a running add-on isn't asked after
|
||||
assert [p["title"] for p in connection.health["problems"]] == ["Something's wrong", "B should start on its own and isn't running."]
|
||||
assert metrics["problems_open"] == 2.0 and metrics["disk_free_pct"] == 50.0 and metrics["update_failed"] == 0.0
|
||||
|
||||
|
||||
async def _no_targets(app):
|
||||
T = _prom("models").PrometheusTarget
|
||||
def test_a_report_it_couldnt_read_leaves_the_last_one_and_home_assistant_still_answers(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
def unreachable(url, **options):
|
||||
raise OSError("connection refused")
|
||||
|
||||
async def go():
|
||||
await _clean(app, health={"repairs_read": True, "problems": [{"title": "Old"}]})
|
||||
await sched.poll(app, transport=ha.transport, connect=unreachable)
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert connection.last_error is None and not connection.refused
|
||||
assert connection.health["problems"] == [{"title": "Old"}] and "connection refused" in connection.health["error"]
|
||||
|
||||
|
||||
def test_a_refused_websocket_stops_the_polling_like_any_refusal(app):
|
||||
sched = _ha("scheduler")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}, auth="auth_invalid"))
|
||||
return (await _state(app))[0]
|
||||
|
||||
connection = asyncio.run(go())
|
||||
assert connection.refused and "Sign in again" in connection.last_error
|
||||
|
||||
|
||||
# ── Updates on a schedule ────────────────────────────────────────────────────
|
||||
|
||||
CORE = "update.home_assistant_core_update"
|
||||
|
||||
|
||||
async def _a_slot_just_passed(app, minutes=10):
|
||||
from datetime import timedelta
|
||||
m = _ha("models")
|
||||
at = datetime.now(timezone.utc) - timedelta(minutes=minutes)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(delete(T).where(T.url.like("%ha.test%") | (T.name == "HomeAssistant")))
|
||||
s.add(m.HomeAssistantSchedule(enabled=True, weekday=at.weekday(), at_time=f"{at:%H:%M}", time_zone="UTC",
|
||||
window_minutes=240, created_at=at - timedelta(days=30)))
|
||||
|
||||
|
||||
async def _the_target(app):
|
||||
T = _prom("models").PrometheusTarget
|
||||
async def _schedule_row(app):
|
||||
m = _ha("models")
|
||||
async with app.db_sessionmaker() as s:
|
||||
return (await s.execute(select(T).where(T.url.like("%ha.test%")))).scalars().first()
|
||||
return (await s.execute(select(m.HomeAssistantSchedule))).scalars().first()
|
||||
|
||||
|
||||
def test_reading_its_metrics_makes_a_target_that_scrapes_with_the_sign_in(app, signed_in):
|
||||
sched = _prom("scheduler")
|
||||
sent = []
|
||||
|
||||
async def fake_scrape(url, token, verify_tls):
|
||||
sent.append((url, token))
|
||||
return await _prom("scraper").scrape(url, token, verify_tls)
|
||||
def test_a_slot_installs_the_add_on_first_then_core_and_stops_at_a_failure(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
m = _ha("models")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.entities[CORE] = ha._update("on", "1.0")
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _no_targets(app)
|
||||
web, _ = await _admin_client(app)
|
||||
page = await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
assert "Read its metrics" in page
|
||||
response = await web.post("/plugins/homeassistant/metrics", headers=CSRF)
|
||||
assert response.status_code == 302
|
||||
await sched.scrape_due(app, scrape=fake_scrape)
|
||||
page = await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
assert "Read with this sign-in" in page
|
||||
return await _the_target(app)
|
||||
|
||||
target = asyncio.run(go())
|
||||
assert target.name == "HomeAssistant" and target.token == "" and target.token_from == "homeassistant"
|
||||
assert "python_*" in target.drop and target.last_error is None and target.last_ok_at
|
||||
assert sent == [(METRICS_URL, "access-1")]
|
||||
|
||||
|
||||
def test_using_the_sign_in_moves_a_target_and_keeps_its_name_and_filters(app, signed_in):
|
||||
from steward.core.crypto import encrypt_secret
|
||||
T = _prom("models").PrometheusTarget
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _no_targets(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport) # the slot fires and asks for a fresh read
|
||||
assert not (await _state(app))[1] and (await _state(app))[0].last_polled_at is None
|
||||
await plan.step(app, ha.transport) # nothing starts before that read
|
||||
assert not (await _state(app))[1]
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
[first] = (await _state(app))[1]
|
||||
assert first.entity_id == ADDON and first.started_by == "schedule"
|
||||
await plan.step(app, ha.transport) # one at a time
|
||||
assert len((await _state(app))[1]) == 1
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(T(name="my-house", url=METRICS_URL, token=encrypt_secret("dead-token"), keep=["a_*"], refused=True))
|
||||
web, _ = await _admin_client(app)
|
||||
page = await (await web.get("/plugins/homeassistant/")).get_data(as_text=True)
|
||||
assert "Use this sign-in" in page
|
||||
assert (await web.post("/plugins/homeassistant/metrics", headers=CSRF)).status_code == 302
|
||||
target = await _the_target(app)
|
||||
await _no_targets(app)
|
||||
return target
|
||||
run = await s.get(m.HomeAssistantUpdate, first.id)
|
||||
run.state, run.message = "failed", "It broke."
|
||||
await plan.step(app, ha.transport)
|
||||
assert "It broke." in (await _schedule_row(app)).stopped
|
||||
await plan.step(app, ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
target = asyncio.run(go())
|
||||
assert (target.name, target.keep) == ("my-house", ["a_*"])
|
||||
assert target.token == "" and target.token_from == "homeassistant" and not target.refused
|
||||
assert [r.entity_id for r in asyncio.run(go())] == [ADDON]
|
||||
assert ha.calls == [BACKUP]
|
||||
|
||||
|
||||
def test_metrics_home_assistant_doesnt_serve_change_nothing_and_say_how(app, signed_in):
|
||||
signed_in.no_metrics = True
|
||||
def test_a_slot_moves_on_after_a_good_update_and_never_retries_one(app):
|
||||
sched, plan, updates = _ha("scheduler"), _ha("schedule"), _ha("updates")
|
||||
ha = FakeHomeAssistant()
|
||||
ha.entities[CORE] = ha._update("on", "1.0")
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _no_targets(app)
|
||||
web, _ = await _admin_client(app)
|
||||
response = await web.post("/plugins/homeassistant/metrics", headers=CSRF)
|
||||
return response.status_code, await response.get_data(as_text=True), await _the_target(app)
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
ha.backup_lands()
|
||||
await updates.advance(app, ha.transport) # the backup landed
|
||||
await updates.advance(app, ha.transport) # the install is sent
|
||||
ha.update_lands()
|
||||
await updates.advance(app, ha.transport) # it's done, and asks for a fresh read
|
||||
await plan.step(app, ha.transport)
|
||||
assert len((await _state(app))[1]) == 1
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
runs = (await _state(app))[1]
|
||||
return {r.entity_id: r.state for r in runs}
|
||||
|
||||
status, page, target = asyncio.run(go())
|
||||
assert status == 409 and "configuration.yaml" in page and target is None
|
||||
assert asyncio.run(go()) == {ADDON: "done", CORE: "backing_up"}
|
||||
|
||||
|
||||
def test_a_refused_scrape_stops_the_sign_in_and_try_again_starts_both(app, signed_in):
|
||||
sched, metrics = _prom("scheduler"), _ha("metrics")
|
||||
TokenRefused = _prom("scraper").TokenRefused
|
||||
T = _prom("models").PrometheusTarget
|
||||
asked = []
|
||||
|
||||
async def refusing(url, token, verify_tls):
|
||||
asked.append(url)
|
||||
raise TokenRefused("Answered 401: the token was refused.")
|
||||
|
||||
async def overdue():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
row = await s.get(T, (await _the_target(app)).id)
|
||||
row.last_scraped_at = row.last_scraped_at.replace(year=2000)
|
||||
def test_a_slot_that_cant_start_an_update_says_why_and_asks_no_more(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _no_targets(app)
|
||||
web, _ = await _admin_client(app)
|
||||
await web.post("/plugins/homeassistant/metrics", headers=CSRF)
|
||||
await sched.scrape_due(app, scrape=refusing)
|
||||
connection = (await _state(app))[0]
|
||||
assert connection.refused and connection.last_error == metrics.REFUSED
|
||||
await _a_slot_just_passed(app)
|
||||
await plan.step(app, ha.transport)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
del ha.entities["sensor.backup_last_successful_automatic_backup"]
|
||||
await plan.step(app, ha.transport)
|
||||
return await _schedule_row(app)
|
||||
|
||||
# Nothing more is sent, by the scrape or the poll, and the target says why.
|
||||
await overdue()
|
||||
await sched.scrape_due(app, scrape=refusing)
|
||||
assert len(asked) == 1
|
||||
target = await _the_target(app)
|
||||
assert not target.refused and "Try again" in target.last_error
|
||||
|
||||
assert (await web.post("/plugins/homeassistant/connection/retry", headers=CSRF)).status_code == 302
|
||||
await overdue()
|
||||
await sched.scrape_due(app, scrape=refusing)
|
||||
assert len(asked) == 2
|
||||
await _no_targets(app)
|
||||
|
||||
asyncio.run(go())
|
||||
row = asyncio.run(go())
|
||||
assert "wasn't started" in row.stopped and "no automatic backups" in row.stopped and not ha.calls
|
||||
|
||||
|
||||
def test_the_sign_in_isnt_sent_to_another_address(app, signed_in):
|
||||
sched = _prom("scheduler")
|
||||
T = _prom("models").PrometheusTarget
|
||||
asked = []
|
||||
|
||||
async def fake_scrape(url, token, verify_tls):
|
||||
asked.append(url)
|
||||
return {}
|
||||
def test_a_slot_past_its_window_starts_nothing_and_a_new_schedule_waits_for_its_next(app):
|
||||
sched, plan = _ha("scheduler"), _ha("schedule")
|
||||
ha = FakeHomeAssistant()
|
||||
|
||||
async def go():
|
||||
await _clean(app)
|
||||
await _no_targets(app)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(T(name="elsewhere", url="http://elsewhere.ha.test/metrics", token_from="homeassistant"))
|
||||
await sched.scrape_due(app, scrape=fake_scrape)
|
||||
target = await _the_target(app)
|
||||
await _no_targets(app)
|
||||
return target
|
||||
await _a_slot_just_passed(app, minutes=300)
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
assert not (await _state(app))[1]
|
||||
await _clean(app)
|
||||
at = datetime.now(timezone.utc)
|
||||
await plan.save(app, enabled=True, weekday=at.weekday(), at_time="00:00", time_zone="UTC")
|
||||
await sched.poll(app, transport=ha.transport, connect=FakeSocket({}))
|
||||
await plan.step(app, ha.transport)
|
||||
return (await _state(app))[1]
|
||||
|
||||
target = asyncio.run(go())
|
||||
assert asked == [] and "isn't the address" in target.last_error and signed_in.refreshes == []
|
||||
assert asyncio.run(go()) == []
|
||||
assert ha.calls == []
|
||||
|
||||
@@ -4,7 +4,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace as NS
|
||||
|
||||
import plugins.homeassistant as homeassistant_plugin
|
||||
from plugins.homeassistant import mcp_tools, scheduler, service as s
|
||||
from plugins.homeassistant import mcp_tools, schedule, scheduler, service as s
|
||||
from steward.models.users import UserRole
|
||||
from tests._templates import render_plugin_template
|
||||
from tests.test_style_tokens import hex_colours
|
||||
@@ -86,7 +86,7 @@ def _connection(**over):
|
||||
"verify_tls": True, "signed_in_by": "someone", "signed_in_at": NOW,
|
||||
"refused": False, "last_error": None, "last_polled_at": NOW, "last_ok_at": NOW,
|
||||
"backup": {"known": True, "last_ok": NOW.isoformat(), "last_result": "completed"},
|
||||
"updates": [], **over})
|
||||
"updates": [], "health": {}, **over})
|
||||
|
||||
|
||||
def test_the_overview_names_what_needs_a_look_and_no_more():
|
||||
@@ -101,6 +101,50 @@ def test_the_overview_names_what_needs_a_look_and_no_more():
|
||||
failed = NS(state="failed", title="Core", message="No new backup.", finished_at=NOW)
|
||||
[f] = mcp_tools.overview_findings(_connection(), failed, NOW)
|
||||
assert f["kind"] == "home_assistant_update_failed" and "No new backup." in f["summary"]
|
||||
two = {"problems": [{"title": "Disk trouble"}, {"title": "Another"}]}
|
||||
[f] = mcp_tools.overview_findings(_connection(health=two), None, NOW)
|
||||
assert f["kind"] == "home_assistant_problem" and f["summary"] == "Disk trouble (and 1 more)"
|
||||
plan = NS(stopped="Core: it failed.", last_slot_at=None)
|
||||
[f] = mcp_tools.overview_findings(_connection(), None, NOW, plan)
|
||||
assert f["kind"] == "home_assistant_schedule_stopped" and "Core: it failed." in f["summary"]
|
||||
|
||||
|
||||
def test_system_problems_are_read_from_what_home_assistant_answers():
|
||||
healthy = s.read_health({"config": {"recovery_mode": False}, "issues": [], "resolution": {"unhealthy": [], "unsupported": []},
|
||||
"host": {"disk_free": 99.8, "disk_total": 116.5}, "addons": [{"slug": "a", "state": "started"}]})
|
||||
assert healthy["problems"] == [] and healthy["repairs_read"] and healthy["supervised"]
|
||||
assert healthy["disk_free_pct"] == 85.7
|
||||
bad = s.read_health({
|
||||
"config": {"recovery_mode": True},
|
||||
"issues": [{"domain": "hassio", "translation_key": "issue_system_reboot_required", "severity": "warning"},
|
||||
{"domain": "demo", "translation_key": "named", "translation_placeholders": {"who": "X"}, "severity": "critical"},
|
||||
{"domain": "demo", "translation_key": "gone", "ignored": True},
|
||||
{"domain": "demo", "translation_key": "old", "dismissed_version": "1.0"}],
|
||||
"titles": {"component.demo.issues.named.title": "{who} broke"},
|
||||
"resolution": {"unhealthy": ["docker_gateway"], "unsupported": []},
|
||||
"host": {"disk_free": 5.0, "disk_total": 100.0},
|
||||
"addons": [{"slug": "should", "name": "Should run", "state": "stopped"}, {"slug": "once", "state": "stopped"}],
|
||||
"boots": {"should": "auto", "once": "manual"}})
|
||||
assert [p["kind"] for p in bad["problems"]] == ["recovery", "repair", "repair", "supervisor", "disk", "addon"]
|
||||
titles = [p["title"] for p in bad["problems"]]
|
||||
assert titles[1] == "System reboot required" and titles[2] == "X broke" and "docker gateway" in titles[3]
|
||||
assert "5 GB of 100 GB" in titles[4] and "Should run" in titles[5]
|
||||
|
||||
|
||||
def test_what_home_assistant_wouldnt_answer_is_not_read_as_fine():
|
||||
found = s.read_health({"config": {}, "issues": None, "resolution": None, "host": None, "addons": None})
|
||||
assert not found["repairs_read"] and not found["supervised"] and found["disk_free_pct"] is None
|
||||
|
||||
|
||||
def test_a_schedule_installs_add_ons_then_core_then_the_os():
|
||||
def update(entity_id, kind, in_progress=False):
|
||||
return {"entity_id": entity_id, "kind": kind, "in_progress": in_progress}
|
||||
waiting = [update("os", "os"), update("core", "core"), update("addon", "addon")]
|
||||
assert schedule.next_to_install(waiting, set())["entity_id"] == "addon"
|
||||
assert schedule.next_to_install(waiting, {"addon"})["entity_id"] == "core"
|
||||
assert schedule.next_to_install(waiting, {"addon", "core"})["entity_id"] == "os"
|
||||
assert schedule.next_to_install(waiting, {"addon", "core", "os"}) is None
|
||||
assert schedule.next_to_install([update("addon", "addon", in_progress=True)], set()) is None
|
||||
|
||||
|
||||
def test_reading_is_open_to_a_viewer_and_installing_is_for_an_admin():
|
||||
@@ -109,7 +153,8 @@ def test_reading_is_open_to_a_viewer_and_installing_is_for_an_admin():
|
||||
def register(fn, *, role, read_only): # both required, as on the server
|
||||
seen[fn.__name__] = (role, read_only)
|
||||
homeassistant_plugin.register_mcp_tools(register)
|
||||
assert seen == {"get_home_assistant": (UserRole.viewer, True), "update_home_assistant": (UserRole.admin, False)}
|
||||
assert seen == {"get_home_assistant": (UserRole.viewer, True), "update_home_assistant": (UserRole.admin, False),
|
||||
"set_home_assistant_schedule": (UserRole.admin, False)}
|
||||
|
||||
|
||||
def _page_values(**over):
|
||||
@@ -119,7 +164,9 @@ def _page_values(**over):
|
||||
return {"connection": _connection(), "reach": "up", "says": "Answering.", "installable": [update], "others": [],
|
||||
"current": [], "runs": [], "under_way": None, "backup": {}, "backup_tone": "ok",
|
||||
"backup_says": "Backups are up to date.", "backup_at": NOW, "backup_next": None, "backup_age": 1.0,
|
||||
"stale_hours": 26, **over}
|
||||
"stale_hours": 26, "health": {"repairs_read": True, "supervised": True}, "problems": [], "plan": None,
|
||||
"plan_when": None, "plan_next": None, "weekdays": ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
|
||||
"Saturday", "Sunday"), "default_zone": "UTC", **over}
|
||||
|
||||
|
||||
async def _render(app, role, **values):
|
||||
@@ -151,6 +198,30 @@ async def test_install_is_held_back_while_it_couldnt_work_and_the_page_says_why(
|
||||
assert "isn't an administrator" in status_only
|
||||
|
||||
|
||||
async def test_the_page_shows_system_problems_and_what_it_couldnt_read(app):
|
||||
clear = await _render(app, "viewer", **_page_values())
|
||||
assert "No repairs open" in clear
|
||||
problem = {"kind": "repair", "severity": "warning", "title": "System reboot required"}
|
||||
shown = await _render(app, "viewer", **_page_values(problems=[problem]))
|
||||
assert "System reboot required" in shown and "Settings → System → Repairs" in shown
|
||||
hidden = await _render(app, "viewer", **_page_values(health={"repairs_read": False, "supervised": False}))
|
||||
assert "wouldn't share its repairs" in hidden
|
||||
bare = await _render(app, "viewer", **_page_values(health={"repairs_read": True, "supervised": False}))
|
||||
assert "No Supervisor answered" in bare
|
||||
|
||||
|
||||
async def test_an_admin_sets_the_schedule_warned_of_what_it_does_and_a_viewer_only_reads_it(app):
|
||||
off = await _render(app, "admin", **_page_values())
|
||||
assert 'action="/plugins/homeassistant/schedule"' in off and "Off. Updates are only installed" in off
|
||||
assert "with nobody there to watch" in off
|
||||
assert '<option value="5" selected>Saturday</option>' in off and 'value="05:00"' in off
|
||||
plan = NS(enabled=True, weekday=5, at_time="05:00", time_zone="Europe/London", stopped="Core: it failed.")
|
||||
values = _page_values(plan=plan, plan_when="Saturdays at 05:00 (Europe/London), up to 4 h", plan_next=NOW)
|
||||
on = await _render(app, "viewer", **values)
|
||||
assert "Saturdays at 05:00" in on and "Core: it failed." in on
|
||||
assert 'action="/plugins/homeassistant/schedule"' not in on
|
||||
|
||||
|
||||
async def test_an_unconnected_page_offers_to_connect(app):
|
||||
page = await _render(app, "admin", connection=None)
|
||||
assert 'href="/plugins/homeassistant/connection"' in page
|
||||
|
||||
@@ -141,10 +141,6 @@ def test_the_form_checks_what_it_stores_and_keeps_a_blank_token():
|
||||
assert again["token"] == "enc:v1:kept" and "enabled" not in again
|
||||
cleared = parse_form(_Form(name="Archives", url="http://a/metrics", clear_token="on"), existing)
|
||||
assert cleared["token"] == ""
|
||||
lent = PrometheusTarget(name="Archives", url="http://a/metrics", token="", token_from="a_plugin")
|
||||
assert parse_form(_Form(name="Archives", url="http://a/metrics"), lent)["token_from"] == "a_plugin"
|
||||
typed = parse_form(_Form(name="Archives", url="http://a/metrics", token="s3cret"), lent)
|
||||
assert typed["token_from"] is None and typed["token"]
|
||||
for bad in (dict(name="", url="http://a"), dict(name="a:b", url="http://a"),
|
||||
dict(name="a", url="ftp://a"), dict(name="a", url="http://a", interval_seconds="5"),
|
||||
dict(name="a", url="http://a", series_cap="0")):
|
||||
|
||||
Reference in New Issue
Block a user