diff --git a/docs/plugins/homeassistant.md b/docs/plugins/homeassistant.md index 31ac24b..1cef2cd 100644 --- a/docs/plugins/homeassistant.md +++ b/docs/plugins/homeassistant.md @@ -1,6 +1,6 @@ # Home Assistant plugin -Shows which Home Assistant updates are waiting and how old its last backup is, and installs Core, OS and add-on updates, each only after a fresh backup is confirmed. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API. +Keeps a Home Assistant instance running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates, on a click or a weekly schedule, each only after a fresh backup is confirmed. It looks after Home Assistant itself and doesn't read your devices. Each update goes on the timeline. Nothing is added to Home Assistant: you sign in, and Steward uses its standard API. ## Setting it up @@ -12,7 +12,7 @@ Your browser and Steward both need to reach Home Assistant at that address. Home ## What Steward holds -**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status and metrics, take backups and install updates. It is stored encrypted. +**Steward stores a credential with the same access as the user who signed in.** Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates. It is stored encrypted. We recommend a service account, a user made just for Steward, rather than your own. What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do. @@ -29,10 +29,22 @@ Install asks once, then Steward: One runs at a time, and it carries on if Steward restarts. Firmware and custom integrations are left to Home Assistant, and Supervisor updates itself. +## System problems + +Each read also asks Home Assistant whether the system itself has a problem, and the page and the overview show what it says: + +- open repairs, from Settings → System → Repairs +- the Supervisor calling the system unhealthy or unsupported +- less than 10% of the disk free +- starting in recovery mode, when its configuration didn't load +- an add-on set to start on its own that isn't running + +Repairs need an administrator's sign-in. Disk space and add-ons come from the Supervisor, so an install without one doesn't report them, and the page says so. + +## Scheduled updates + +Set a day and time under Scheduled updates on the page. It's off until you do. At that time Steward installs what's waiting, add-ons first, then Core, then the OS, one at a time and each after its own confirmed backup. It starts nothing new more than 4 hours after the slot. The first failure ends the run and the rest wait, and the page, the overview and `update_failed` say so. A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch, and an update can change how automations behave. + ## Metrics -One sign-in covers everything: the Metrics section of the page reads Home Assistant's entity readings with the same user, through a target on the Prometheus page. **Read its metrics** makes that target, or **Use this sign-in** moves an existing one at that address onto it, keeping its name, filters and history and forgetting its token. Home Assistant only serves these with its Prometheus integration turned on (`prometheus:` in its `configuration.yaml`, then a restart). Steward doesn't change that for you; it says so if they aren't there. - -If Home Assistant refuses that user, everything stops after one refusal, and signing in again or Try again starts everything. To go back to a separate token, type one into the target's form on the Prometheus page. - -The poll itself records these under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting` and `backup_age_hours`. An alert rule on `backup_age_hours` above 26 warns when a daily backup has been missed; a rule's no-data warning covers Home Assistant going quiet. +Under source `homeassistant`, resource `homeassistant`, read every 5 minutes: `is_up`, `updates_waiting`, `backup_age_hours`, `problems_open`, `disk_free_pct` and `update_failed`. Useful alert rules: `backup_age_hours` above 26 for a missed daily backup, `problems_open` above 0, and `update_failed` above 0. A rule's no-data warning covers Home Assistant going quiet. diff --git a/docs/plugins/prometheus.md b/docs/plugins/prometheus.md index 469f090..36df618 100644 --- a/docs/plugins/prometheus.md +++ b/docs/plugins/prometheus.md @@ -48,8 +48,6 @@ A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as **A refused token stops the scraping.** When an endpoint answers 401 or 403, Steward stops asking and says so on the target's page and in the overview, because asking again won't change the answer and some services, Home Assistant among them, ban an address after a few refusals. Saving the target, usually with a new token, or pausing and resuming it, starts it again. Any other failure is retried at the next scrape. -**A target can use a plugin's sign-in instead of a token.** Home Assistant's page sets this up, so one Home Assistant user covers its metrics too. A token typed into the target's form replaces it. That sign-in is only sent to the address it was made for, and a refusal stops it from the plugin's page, not here. - A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state. **Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were. diff --git a/plugins/homeassistant/__init__.py b/plugins/homeassistant/__init__.py index 50dbea9..b4af7d7 100644 --- a/plugins/homeassistant/__init__.py +++ b/plugins/homeassistant/__init__.py @@ -11,7 +11,7 @@ _app: "Quart | None" = None def setup(app: "Quart") -> None: global _app _app = app - from .models import HomeAssistantConnection, HomeAssistantUpdate # noqa: F401 — registers the tables + from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate # noqa: F401 — registers the tables from .timeline import register register() @@ -32,7 +32,8 @@ def get_nav() -> list[dict]: def get_alert_metrics() -> dict[str, list[str]]: """What the poll records, all under the resource "homeassistant".""" - return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting"]} + return {"homeassistant": ["is_up", "backup_age_hours", "updates_waiting", "problems_open", "disk_free_pct", + "update_failed"]} def register_mcp_tools(register) -> None: diff --git a/plugins/homeassistant/client.py b/plugins/homeassistant/client.py index 813dff1..6baacc8 100644 --- a/plugins/homeassistant/client.py +++ b/plugins/homeassistant/client.py @@ -1,17 +1,20 @@ # plugins/homeassistant/client.py """Talking to Home Assistant, with what its own sign-in gave Steward: a refresh token kept in the connection, and access tokens made from it that last half an -hour. Home Assistant bans an address after a few refused requests, so nothing is +hour. Repairs and the Supervisor's reports are only offered on its websocket. Home Assistant bans an address after a few refused requests, so nothing is sent without a token and a refusal stops the asking. Every failure comes back as one sentence a page can show.""" from __future__ import annotations import asyncio import json +import ssl import time from urllib.parse import urlencode import httpx +from websockets.asyncio.client import connect as ws_connect +from websockets.exceptions import WebSocketException from steward.core.crypto import decrypt_secret @@ -23,6 +26,8 @@ ACTION_READ_SECONDS = 10.0 MAX_BYTES = 16 * 1024 * 1024 # An access token this close to running out isn't used for another request. EXPIRY_MARGIN_SECONDS = 60 +# One read of the system over the websocket, however its answers dribble in. +SYSTEM_READ_SECONDS = 30 _access: dict[str, tuple[str, float]] = {} # refresh token -> access token, when to stop using it @@ -118,12 +123,6 @@ async def redeem_code(url: str, client_id: str, code: str, verify_tls: bool, tra return refresh_token, _remember(refresh_token, found) -def forget_access() -> None: - """Drop access tokens made so far: after a refusal the next ask starts from the - sign-in, which a dead user is refused at without counting toward a ban.""" - _access.clear() - - async def access_token(connection, transport=None) -> str: refresh_token = decrypt_secret(connection.refresh_token, context="Home Assistant sign-in") token, until = _access.get(refresh_token, (None, 0.0)) @@ -214,3 +213,65 @@ async def call_action(url: str, domain: str, action: str, data: dict, token: str except (ValueError, AttributeError): said = None raise Rejected(said or f"Home Assistant answered {status}.") + + +# ── The system, over the websocket ─────────────────────────────────────────── + +class _Socket: + """Home Assistant's websocket after sign-in: one question at a time, each with + an id, and None for one it won't answer (no Supervisor, or not an administrator).""" + + def __init__(self, ws): + self.ws, self.asked = ws, 0 + + async def sign_in(self, token: str) -> None: + await self.ws.recv() + await self.ws.send(json.dumps({"type": "auth", "access_token": token})) + if json.loads(await self.ws.recv()).get("type") != "auth_ok": + raise Refused("Home Assistant refused Steward's sign-in. Sign in again, as a user who may do this.") + + async def ask(self, **question): + self.asked += 1 + await self.ws.send(json.dumps({"id": self.asked, **question})) + while True: + answer = json.loads(await self.ws.recv()) + if answer.get("id") == self.asked: + return answer.get("result") if answer.get("success") else None + + async def supervisor(self, endpoint: str): + return await self.ask(type="supervisor/api", endpoint=endpoint, method="get") + + +async def read_system(url: str, token: str, verify_tls: bool, connect=None) -> dict: + """What service.read_health needs, as Home Assistant answered it. `connect` + stands in for the websocket in tests.""" + target = "ws" + url.rstrip("/").removeprefix("http") + "/api/websocket" + options: dict = {"max_size": MAX_BYTES} + if target.startswith("wss:") and not verify_tls: + options["ssl"] = ssl.create_default_context() + options["ssl"].check_hostname, options["ssl"].verify_mode = False, ssl.CERT_NONE + try: + async with asyncio.timeout(SYSTEM_READ_SECONDS): + async with (connect or ws_connect)(target, **options) as ws: + socket = _Socket(ws) + await socket.sign_in(token) + found = {"config": await socket.ask(type="get_config"), + "issues": (await socket.ask(type="repairs/list_issues") or {}).get("issues"), + "resolution": await socket.supervisor("/resolution/info"), + "host": await socket.supervisor("/host/info"), "titles": {}, "boots": {}} + domains = sorted({i["domain"] for i in found["issues"] or []}) + if domains: + titles = await socket.ask(type="frontend/get_translations", language="en", + category="issues", integration=domains) + found["titles"] = (titles or {}).get("resources") or {} + found["addons"] = (await socket.supervisor("/addons") or {}).get("addons") + # Whether one should be running is only on its own page, so a running one isn't asked. + for addon in found["addons"] or []: + if addon.get("state") != "started": + info = await socket.supervisor(f"/addons/{addon['slug']}/info") or {} + found["boots"][addon["slug"]] = info.get("boot") + return found + except TimeoutError: + raise Unreachable(f"Home Assistant's system report didn't arrive within {SYSTEM_READ_SECONDS} s.") from None + except (OSError, ValueError, KeyError, WebSocketException) as exc: + raise Unreachable(f"Couldn't read Home Assistant's system report: {exc}") from None diff --git a/plugins/homeassistant/mcp_tools.py b/plugins/homeassistant/mcp_tools.py index 358ee48..63bd698 100644 --- a/plugins/homeassistant/mcp_tools.py +++ b/plugins/homeassistant/mcp_tools.py @@ -5,23 +5,35 @@ update as the page's Install does. The sign-in Steward holds is never returned. """ from __future__ import annotations +from datetime import timezone + from sqlalchemy import select +from steward.core import weekly +from steward.core.formatting import zone from steward.mcp._context import current_caller, mcp_app from steward.mcp.tools._audit import audit_mcp, mcp_username from steward.mcp.tools._shape import age_s, db_session, iso, now_utc, redact_url from steward.mcp.tools.overview import finding from steward.models.users import UserRole -from . import metrics -from .models import HomeAssistantConnection, HomeAssistantUpdate +from . import schedule +from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate from .routes import connection_says -from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment +from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open from .updates import CannotStart, start_update RECENT_UPDATES = 10 +def _schedule_view(plan, now) -> dict | None: + if plan is None: + return None + return {"enabled": plan.enabled, "when": weekly.describe(plan), + "next_slot_at": iso(weekly.next_slot(plan, now)) if plan.enabled else None, + "last_slot_at": iso(weekly.fired_slot(plan)), "stopped": plan.stopped} + + def _run_view(r: HomeAssistantUpdate) -> dict: return {"update_id": r.id, "entity_id": r.entity_id, "title": r.title, "kind": r.kind, "from_version": r.from_version, "to_version": r.to_version, "state": r.state, "message": r.message, @@ -36,16 +48,17 @@ async def get_home_assistant() -> dict: `installable` is what update_home_assistant takes (Core, OS, add-ons); `others_waiting` is firmware and custom integrations, which Steward leaves to Home Assistant. `can_install` is whether the Home Assistant user who signed - Steward in is an administrator, which installing needs. `metrics` is the - Prometheus target that reads its metrics, `uses_sign_in` saying whether that - same user does; null when the Prometheus plugin is off. + Steward in is an administrator, which installing needs. `system.problems` is + what's wrong with the system itself: open repairs, Supervisor health, low disk, + recovery mode, add-ons that should be running. `schedule` is the weekly slot + for installing what's waiting, null until one is set. """ now = now_utc() async with db_session() as db: c = (await db.execute(select(HomeAssistantConnection))).scalars().first() runs = list((await db.execute(select(HomeAssistantUpdate).order_by( HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars()) - scrape = await metrics.state(db, c) if c else None + plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first() if c is None: return {"connected": False, "says": "No connection yet: an admin signs in on the Home Assistant page."} reach, says = connection_says(c, now) @@ -59,7 +72,8 @@ async def get_home_assistant() -> dict: "backup": {"verdict": backup_verdict(backup, now)[1], "last_ok": iso(parse_moment(backup.get("last_ok"))), "age_hours": round(age, 1) if age is not None else None, "stale_after_hours": STALE_BACKUP_HOURS, "next": iso(parse_moment(backup.get("next"))), "last_result": backup.get("last_result")}, - "metrics": scrape, "updates": [_run_view(r) for r in runs]} + "system": c.health or {}, "schedule": _schedule_view(plan, now), + "updates": [_run_view(r) for r in runs]} async def update_home_assistant(entity_id: str) -> dict: @@ -82,9 +96,35 @@ async def update_home_assistant(entity_id: str) -> dict: return {**_run_view(run), "next": "get_home_assistant follows it: `updates` shows its state."} -def overview_findings(c, latest_run, now) -> list[dict]: - """A connection that isn't answering, stale backups, and a newest update that - failed. Updates waiting aren't a problem to raise.""" +async def set_home_assistant_schedule(enabled: bool, weekday: int = 5, at_time: str = "05:00", + time_zone: str = "UTC") -> dict: + """Set the weekly slot for installing what's waiting, or turn it off, as the page does. + + weekday: 0 for Monday to 6 for Sunday. at_time: "HH:MM" in time_zone. + + At each slot Steward installs add-ons, then Core, then the OS, one at a time, + each after a confirmed backup, stopping at the first failure. A Core update + restarts Home Assistant and an OS update reboots the machine, with nobody + watching. The first slot is the next one from now. Admin only, and audited. + """ + try: + weekly.parse_at(at_time) + except ValueError: + raise ValueError(f"at_time is HH:MM, not {at_time!r}.") from None + if not 0 <= weekday <= 6: + raise ValueError("weekday is 0 for Monday to 6 for Sunday.") + if zone(time_zone) is timezone.utc and time_zone.upper() != "UTC": + raise ValueError(f"Unknown time zone {time_zone!r}.") + row = await schedule.save(mcp_app(), enabled=enabled, weekday=weekday, at_time=at_time, time_zone=time_zone) + await audit_mcp("homeassistant.schedule_saved", "homeassistant_schedule", row.id, + {"enabled": row.enabled, "when": weekly.describe(row)}) + return _schedule_view(row, now_utc()) + + +def overview_findings(c, latest_run, now, plan=None) -> list[dict]: + """A connection that isn't answering, a problem with the system, stale backups, + a newest update that failed, and a scheduled run that stopped. Updates waiting + aren't a problem to raise.""" if c is None: return [] found = [] @@ -92,6 +132,11 @@ def overview_findings(c, latest_run, now) -> list[dict]: if reach == "down": found.append(finding("warning", "home_assistant_down", "Home Assistant", f"Home Assistant isn't answering: {says}", since=c.last_ok_at, next="get_home_assistant()")) + problems = problems_open(c.health) + if problems: + more = f" (and {len(problems) - 1} more)" if len(problems) > 1 else "" + found.append(finding("warning", "home_assistant_problem", "Home Assistant", f"{problems[0]['title']}{more}", + next="get_home_assistant()")) tone, backup_says = backup_verdict(c.backup, now) if c.last_ok_at is not None and tone == "warn": found.append(finding("warning", "home_assistant_backup", "Home Assistant", backup_says, @@ -100,6 +145,10 @@ def overview_findings(c, latest_run, now) -> list[dict]: found.append(finding("warning", "home_assistant_update_failed", latest_run.title, f"The update of {latest_run.title} failed: {latest_run.message}", since=latest_run.finished_at, next="get_home_assistant()")) + if plan is not None and plan.stopped: + found.append(finding("warning", "home_assistant_schedule_stopped", "Home Assistant", + f"The scheduled update run stopped: {plan.stopped}", since=weekly.fired_slot(plan), + next="get_home_assistant()")) return found @@ -107,9 +156,11 @@ async def mcp_overview(db) -> list[dict]: c = (await db.execute(select(HomeAssistantConnection))).scalars().first() latest = (await db.execute(select(HomeAssistantUpdate).order_by( HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first() - return overview_findings(c, latest, now_utc()) + plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first() + return overview_findings(c, latest, now_utc(), plan) def register_tools(register) -> None: register(get_home_assistant, role=UserRole.viewer, read_only=True) register(update_home_assistant, role=UserRole.admin, read_only=False) + register(set_home_assistant_schedule, role=UserRole.admin, read_only=False) diff --git a/plugins/homeassistant/metrics.py b/plugins/homeassistant/metrics.py deleted file mode 100644 index b199f2a..0000000 --- a/plugins/homeassistant/metrics.py +++ /dev/null @@ -1,153 +0,0 @@ -# plugins/homeassistant/metrics.py -"""Home Assistant's metrics, read with the one sign-in Steward already holds, so a -single Home Assistant user covers everything. The scrape stays a Prometheus target: -this lends it a token at each scrape, sends that token only to this Home Assistant, -and stops it when the sign-in has been refused.""" -from __future__ import annotations - -import asyncio -from datetime import datetime, timezone -from typing import TYPE_CHECKING - -from sqlalchemy import or_, select, update - -from steward.core.plugin_manager import loaded_plugin_submodule -from steward.core.prometheus import TokenUnavailable - -from . import client -from .models import HomeAssistantConnection -from .service import SOURCE - -if TYPE_CHECKING: - from quart import Quart - -METRICS_PATH = "/api/prometheus" -NAME = "HomeAssistant" -INTERVAL_SECONDS = 120 -TRY_SECONDS = 30 -# What a new target leaves out: bookkeeping on every entity, a few thousand series -# that say nothing the readings don't. -DROP = ["homeassistant_state_change*", "homeassistant_last_updated_time_seconds", "homeassistant_entity_available", - "homeassistant_entity_info", "homeassistant_area_info", "python_*", "process_*"] -REFUSED = "Home Assistant refused Steward's sign-in while reading its metrics, or has banned this address: look in its ip_bans.yaml." - - -class CannotRead(Exception): - """Why the metrics can't be read with this sign-in, said to the person.""" - - -def metrics_url(connection) -> str: - return connection.url.rstrip("/") + METRICS_PATH - - -# ── Lending the sign-in to a scrape ────────────────────────────────────────── - -async def scrape_token(app: "Quart", url: str) -> str: - """A token for scraping `url`. Raises TokenUnavailable, asking nothing of - anyone, once the sign-in was refused: another refused request counts toward a - ban. A target edited to point elsewhere isn't sent this user's token.""" - async with app.db_sessionmaker() as db: - connection = (await db.execute(select(HomeAssistantConnection))).scalars().first() - if connection is None: - raise TokenUnavailable("Home Assistant isn't connected: sign in on its page to read its metrics.") - if connection.refused: - raise TokenUnavailable(f"{connection.last_error} Steward has stopped asking until you sign in again " - "or choose Try again on the Home Assistant page.") - if url != metrics_url(connection): - raise TokenUnavailable(f"This isn't the address of the Home Assistant Steward signed in to, so its sign-in " - f"isn't sent here. Set the address to {metrics_url(connection)}, or type a token.") - try: - return await client.access_token(connection) - except (client.Refused, client.Unreachable) as exc: - raise TokenUnavailable(str(exc)) from None - - -async def token_refused(app: "Quart") -> None: - """A scrape's token was refused: stop everything using this sign-in.""" - client.forget_access() - async with app.db_sessionmaker() as db: - async with db.begin(): - connection = (await db.execute(select(HomeAssistantConnection))).scalars().first() - if connection is not None: - connection.refused, connection.last_error = True, REFUSED - - -# ── The Metrics section of the page ────────────────────────────────────────── - -def _prometheus(sub: str): - return loaded_plugin_submodule("prometheus", sub) - - -async def _target(db, connection): - """The target using this sign-in, or else the one at this address.""" - T = _prometheus("models").PrometheusTarget - rows = (await db.execute(select(T).where(or_(T.token_from == SOURCE, T.url == metrics_url(connection))))).scalars().all() - return next((t for t in rows if t.token_from == SOURCE), rows[0] if rows else None) - - -async def state(db, connection) -> dict | None: - """What the page says about the metrics. None when the Prometheus plugin is off; - `id` is None when no target reads them yet.""" - service = _prometheus("service") - if service is None or _prometheus("models") is None: - return None - target = await _target(db, connection) - if target is None: - return {"id": None, "name": None, "uses_sign_in": False, "reach": None, "says": None, "kept": 0} - found = service.target_state(target, datetime.now(timezone.utc)) - return {"id": target.id, "name": target.name, "uses_sign_in": target.token_from == SOURCE, - "reach": found.reach, "says": found.says, "kept": target.last_kept} - - -async def try_reading(app: "Quart", connection) -> None: - """Ask for the metrics once, so turning this on says at once if they aren't there.""" - scraper = _prometheus("scraper") - if scraper is None: - raise CannotRead("the Prometheus plugin isn't turned on.") - if connection.refused: - raise CannotRead("Home Assistant has refused this sign-in. Choose Try again or sign in again first.") - try: - token = await client.access_token(connection) - await asyncio.wait_for(scraper.scrape(metrics_url(connection), token, connection.verify_tls), TRY_SECONDS) - except scraper.NoMetricsHere: - raise CannotRead("Home Assistant isn't serving metrics. Add prometheus: to its configuration.yaml, restart it, " - "and come back: Steward doesn't change Home Assistant.") from None - except scraper.TokenRefused: - await token_refused(app) - raise CannotRead(REFUSED) from None - except TimeoutError: - raise CannotRead(f"Home Assistant's metrics didn't arrive within {TRY_SECONDS} s.") from None - except (scraper.ScrapeError, client.Refused, client.Unreachable) as exc: - raise CannotRead(str(exc)) from None - - -async def use_sign_in(app: "Quart", connection) -> tuple[str, str, bool]: - """Make the target, or move the one at this address onto this sign-in, keeping - its name, filters and history. Returns (id, name, whether it was made).""" - models, scheduler = _prometheus("models"), _prometheus("scheduler") - T = models.PrometheusTarget - async with app.db_sessionmaker() as db: - async with db.begin(): - target = await _target(db, connection) - created = target is None - if created: - if (await db.execute(select(T.id).where(T.name == NAME))).first(): - raise CannotRead(f"another Prometheus target is already named {NAME}. Rename it first.") - target = T(name=NAME, interval_seconds=INTERVAL_SECONDS, drop=list(DROP)) - db.add(target) - target.url, target.verify_tls = metrics_url(connection), connection.verify_tls - target.token, target.token_from, target.refused = "", SOURCE, False - target.last_scraped_at = target.last_error = None - await db.flush() - target_id, name = target.id, target.name - scheduler.RATES.forget(target_id) - return target_id, name, created - - -async def follow_sign_in(db, connection) -> None: - """Signing in at a new address takes the scrape using this sign-in with it.""" - models = _prometheus("models") - if models is not None: - T = models.PrometheusTarget - await db.execute(update(T).where(T.token_from == SOURCE).values( - url=metrics_url(connection), verify_tls=connection.verify_tls)) diff --git a/plugins/homeassistant/migrations/versions/homeassistant_003_health.py b/plugins/homeassistant/migrations/versions/homeassistant_003_health.py new file mode 100644 index 0000000..4be566d --- /dev/null +++ b/plugins/homeassistant/migrations/versions/homeassistant_003_health.py @@ -0,0 +1,37 @@ +"""Home Assistant: what the poll reads of the system, and a weekly slot for updates + +Revision ID: homeassistant_003_health +Revises: homeassistant_002_sign_in +Create Date: 2026-11-01 +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "homeassistant_003_health" +down_revision: Union[str, None] = "homeassistant_002_sign_in" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column("homeassistant_connection", + sa.Column("health", sa.JSON, nullable=False, server_default="{}")) + op.create_table( + "homeassistant_schedule", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("enabled", sa.Boolean, nullable=False), + sa.Column("weekday", sa.Integer, nullable=False), + sa.Column("at_time", sa.String(5), nullable=False), + sa.Column("time_zone", sa.String(64), nullable=False), + sa.Column("window_minutes", sa.Integer, nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("last_slot_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("stopped", sa.Text, nullable=True), + ) + + +def downgrade() -> None: + op.drop_table("homeassistant_schedule") + op.drop_column("homeassistant_connection", "health") diff --git a/plugins/homeassistant/models.py b/plugins/homeassistant/models.py index 28b1e8b..7cbd404 100644 --- a/plugins/homeassistant/models.py +++ b/plugins/homeassistant/models.py @@ -3,7 +3,7 @@ from __future__ import annotations from datetime import datetime, timezone -from sqlalchemy import JSON, Boolean, DateTime, Index, String, Text, text +from sqlalchemy import JSON, Boolean, DateTime, Index, Integer, String, Text, text from sqlalchemy.orm import Mapped, mapped_column from steward.core.ids import time_ordered_id @@ -17,7 +17,7 @@ class HomeAssistantConnection(Base): has the same access as the user who signed in, whatever that is: Home Assistant has no narrower permission. `can_install` is whether that user was an administrator, which backups need. The last_* columns and `updates` and - `backup` are the newest poll's outcome, which the pages read. `refused` stops + `backup` and `health` are the newest poll's outcome, which the pages read. `refused` stops all polling until someone signs in again or tries again: Home Assistant bans an address after a few refused requests. """ @@ -37,6 +37,25 @@ class HomeAssistantConnection(Base): last_error: Mapped[str | None] = mapped_column(Text, nullable=True) updates: Mapped[list] = mapped_column(JSON, nullable=False, default=list) backup: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) + health: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) + + +class HomeAssistantSchedule(Base): + """The weekly slot for installing what's waiting; at most one row, none until an + admin sets it. `last_slot_at` is the slot that last fired, and `stopped` says why + that slot ended early. Times are read through steward.core.weekly.""" + __tablename__ = "homeassistant_schedule" + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=time_ordered_id) + enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) + weekday: Mapped[int] = mapped_column(Integer, nullable=False) + at_time: Mapped[str] = mapped_column(String(5), nullable=False) + time_zone: Mapped[str] = mapped_column(String(64), nullable=False) + window_minutes: Mapped[int] = mapped_column(Integer, nullable=False) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc)) + last_slot_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + stopped: Mapped[str | None] = mapped_column(Text, nullable=True) class HomeAssistantUpdate(Base): diff --git a/plugins/homeassistant/plugin.yaml b/plugins/homeassistant/plugin.yaml index 12efeea..845adc7 100644 --- a/plugins/homeassistant/plugin.yaml +++ b/plugins/homeassistant/plugin.yaml @@ -1,7 +1,7 @@ # plugins/homeassistant/plugin.yaml name: homeassistant version: "1.0.0" -description: "Shows which Home Assistant updates are waiting and how old its last backup is, and installs Core, OS and add-on updates after a confirmed backup." +description: "Keeps Home Assistant running: shows whether backups are happening and whether the system has a problem, and installs Core, OS and add-on updates after a confirmed backup, on a click or a weekly schedule." author: "Steward" license: "MIT" min_app_version: "0.1.0" diff --git a/plugins/homeassistant/routes.py b/plugins/homeassistant/routes.py index 1c0b5c9..6322d39 100644 --- a/plugins/homeassistant/routes.py +++ b/plugins/homeassistant/routes.py @@ -13,14 +13,16 @@ from quart import Blueprint, current_app, redirect, render_template, request, se from sqlalchemy import select from steward.auth.middleware import require_role +from steward.core import weekly from steward.core.audit import log_audit from steward.core.crypto import encrypt_secret +from steward.core.formatting import zone from steward.core.settings import public_base_url -from steward.models.users import UserRole +from steward.models.users import User, UserRole -from . import client, metrics -from .models import HomeAssistantConnection, HomeAssistantUpdate -from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment +from . import client, schedule +from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate +from .service import STALE_BACKUP_HOURS, backup_age_hours, backup_verdict, installable, parse_moment, problems_open from .updates import CannotStart, start_update logger = logging.getLogger(__name__) @@ -55,23 +57,29 @@ async def _render_index(error: str | None = None, status: int = 200): connection = await _connection(db) runs = list((await db.execute(select(HomeAssistantUpdate).order_by( HomeAssistantUpdate.started_at.desc()).limit(RECENT_UPDATES))).scalars()) - scrape = await metrics.state(db, connection) if connection else None + plan = (await db.execute(select(HomeAssistantSchedule))).scalars().first() + me = await db.get(User, session.get("user_id")) if session.get("user_id") else None if connection is None: return await render_template("homeassistant/index.html", connection=None, error=error), status + default_zone = (me.time_zone if me and me.time_zone else "UTC") now = datetime.now(timezone.utc) reach, says = connection_says(connection, now) updates = connection.updates or [] backup = connection.backup or {} tone, backup_says = backup_verdict(backup, now) return await render_template( - "homeassistant/index.html", connection=connection, error=error, reach=reach, says=says, metrics=scrape, + "homeassistant/index.html", connection=connection, error=error, reach=reach, says=says, installable=installable(connection), others=[u for u in updates if u["waiting"] and not u["kind"]], current=[u for u in updates if not u["waiting"] and u["kind"]], runs=runs, under_way=next((r for r in runs if r.state in ("backing_up", "installing")), None), backup=backup, backup_tone=tone, backup_says=backup_says, backup_at=parse_moment(backup.get("last_ok")), backup_next=parse_moment(backup.get("next")), - backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS), status + backup_age=backup_age_hours(backup, now), stale_hours=STALE_BACKUP_HOURS, + health=connection.health or {}, problems=problems_open(connection.health), + plan=plan, plan_when=weekly.describe(plan) if plan else None, + plan_next=weekly.next_slot(plan, now) if plan and plan.enabled else None, + weekdays=weekly.WEEKDAYS, default_zone=default_zone), status @homeassistant_bp.get("/") @@ -96,23 +104,23 @@ async def install(): return redirect("/plugins/homeassistant/") -@homeassistant_bp.post("/metrics") +@homeassistant_bp.post("/schedule") @require_role(UserRole.admin) -async def connect_metrics(): - """Read Home Assistant's metrics with this sign-in, after asking for them once - to show they're there. An admin's explicit action, so it may ask.""" - async with current_app.db_sessionmaker() as db: - connection = await _connection(db) - if connection is None: - return redirect("/plugins/homeassistant/") +async def save_schedule(): + """Set the weekly slot for installing what's waiting, or turn it off.""" + form = await request.form + time_zone = (form.get("time_zone") or "UTC").strip() try: - await metrics.try_reading(current_app, connection) - target_id, name, created = await metrics.use_sign_in(current_app, connection) - except metrics.CannotRead as exc: - return await _render_index(f"Nothing changed: {exc}", 409) - await log_audit(current_app, *_actor(), "prometheus.target_created" if created else "prometheus.target_updated", - entity_type="prometheus_target", entity_id=target_id, - detail={"name": name, "token_from": metrics.SOURCE}) + weekday, at_time = int(form.get("weekday", "")), (form.get("at_time") or "").strip() + weekly.parse_at(at_time) + except ValueError: + return await _render_index("Choose a day and a time.", 422) + if not 0 <= weekday <= 6 or (zone(time_zone) is timezone.utc and time_zone.upper() != "UTC"): + return await _render_index(f"Choose a day, and a known time zone rather than “{time_zone}”.", 422) + row = await schedule.save(current_app, enabled=bool(form.get("enabled")), weekday=weekday, + at_time=at_time, time_zone=time_zone) + await log_audit(current_app, *_actor(), "homeassistant.schedule_saved", entity_type="homeassistant_schedule", + entity_id=row.id, detail={"enabled": row.enabled, "when": weekly.describe(row)}) return redirect("/plugins/homeassistant/") @@ -188,7 +196,6 @@ async def sign_in_callback(): connection.can_install, connection.signed_in_by = can_install, username connection.signed_in_at = datetime.now(timezone.utc) connection.refused, connection.last_error, connection.last_polled_at = False, None, None - await metrics.follow_sign_in(db, connection) await db.flush() connection_id = connection.id if previous is not None: @@ -214,9 +221,7 @@ async def _end_sign_in(connection) -> bool: @homeassistant_bp.post("/connection/retry") @require_role(UserRole.admin) async def retry_connection(): - """Ask again after a refusal, once a ban has been dealt with. A metrics scrape - using this sign-in starts again with it.""" - client.forget_access() + """Ask again after a refusal, once a ban has been dealt with.""" async with current_app.db_sessionmaker() as db: async with db.begin(): connection = await _connection(db) diff --git a/plugins/homeassistant/schedule.py b/plugins/homeassistant/schedule.py new file mode 100644 index 0000000..48708e3 --- /dev/null +++ b/plugins/homeassistant/schedule.py @@ -0,0 +1,95 @@ +# plugins/homeassistant/schedule.py +"""Installing what's waiting at a weekly slot: one at a time through the same path +as the page's Install, each after its own confirmed backup. Add-ons go first and +the OS last, as it reboots the machine. The first failure ends the slot, leaving the +rest for someone to look at. Everything is on the rows, so a restart of Steward or +of Home Assistant only delays the next step.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from sqlalchemy import select + +from steward.core import weekly + +from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate +from .service import installable +from .updates import CannotStart, start_update, under_way + +ACTOR = "schedule" +WINDOW_MINUTES = 240 +_ORDER = {"addon": 0, "core": 1, "os": 2} + + +def next_to_install(waiting: list[dict], tried: set[str]) -> dict | None: + left = [u for u in waiting if u["entity_id"] not in tried and not u["in_progress"]] + return min(left, key=lambda u: _ORDER[u["kind"]], default=None) + + +async def _stop(app, schedule_id: str, why: str) -> None: + async with app.db_sessionmaker() as session: + async with session.begin(): + row = await session.get(HomeAssistantSchedule, schedule_id) + if row is not None: + row.stopped = why + + +async def _fire(app, schedule_id: str, slot: datetime) -> None: + """Open the slot, and have the next poll say what's waiting at this moment.""" + async with app.db_sessionmaker() as session: + async with session.begin(): + row = await session.get(HomeAssistantSchedule, schedule_id) + row.last_slot_at, row.stopped = slot, None + connection = (await session.execute(select(HomeAssistantConnection))).scalars().first() + if connection is not None: + connection.last_polled_at = None + + +async def step(app, transport=None) -> None: + """Fire a slot that has come, or start the next update of one that's open.""" + now = datetime.now(timezone.utc) + async with app.db_sessionmaker() as session: + schedule = (await session.execute(select(HomeAssistantSchedule))).scalars().first() + connection = (await session.execute(select(HomeAssistantConnection))).scalars().first() + if schedule is None or connection is None: + return + slot = weekly.is_due(schedule, now) + if slot is not None: + await _fire(app, schedule.id, slot) + return + slot = weekly.fired_slot(schedule) + if not schedule.enabled or schedule.stopped or slot is None \ + or now - slot >= timedelta(minutes=schedule.window_minutes): + return + # Only on a fresh, good read: Home Assistant may be restarting from the last update. + if connection.refused or connection.last_polled_at is None or connection.last_error: + return + if await under_way(session) is not None: + return + runs = (await session.execute(select(HomeAssistantUpdate).where( + HomeAssistantUpdate.started_at >= slot))).scalars().all() + failed = next((r for r in runs if r.state == "failed"), None) + if failed is not None: + await _stop(app, schedule.id, f"{failed.title}: {failed.message} Nothing more was installed in this slot.") + return + update = next_to_install(installable(connection), {r.entity_id for r in runs}) + if update is None: + return + try: + await start_update(app, update["entity_id"], ACTOR, transport) + except CannotStart as exc: + await _stop(app, schedule.id, f"{update['title']} wasn't started: {exc}. Nothing more was installed in this slot.") + + +async def save(app, *, enabled: bool, weekday: int, at_time: str, time_zone: str) -> HomeAssistantSchedule: + """Set the slot. Its first is the next one from now, not one already past.""" + async with app.db_sessionmaker() as session: + async with session.begin(): + row = (await session.execute(select(HomeAssistantSchedule))).scalars().first() + if row is None: + row = HomeAssistantSchedule(window_minutes=WINDOW_MINUTES) + session.add(row) + row.enabled, row.weekday, row.time_zone = enabled, weekday, time_zone + row.at_time = f"{weekly.parse_at(at_time):%H:%M}" + row.last_slot_at, row.stopped = datetime.now(timezone.utc), None + return row diff --git a/plugins/homeassistant/scheduler.py b/plugins/homeassistant/scheduler.py index 221926f..2e50ec7 100644 --- a/plugins/homeassistant/scheduler.py +++ b/plugins/homeassistant/scheduler.py @@ -10,9 +10,9 @@ from sqlalchemy import select from steward.core.scheduler import ScheduledTask -from . import client -from .models import HomeAssistantConnection -from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_snapshot +from . import client, schedule +from .models import HomeAssistantConnection, HomeAssistantSchedule, HomeAssistantUpdate +from .service import RESOURCE, SOURCE, backup_age_hours, installable, read_health, read_snapshot from .updates import advance if TYPE_CHECKING: @@ -36,7 +36,7 @@ def make_tick_task(app: "Quart") -> ScheduledTask: if busy.locked(): return async with busy: - for step in (poll, advance): + for step in (poll, advance, schedule.step): try: await asyncio.wait_for(step(app), STEP_DEADLINE_SECONDS) except TimeoutError: @@ -56,7 +56,15 @@ def is_due(connection, now: datetime) -> bool: return (now - connection.last_polled_at).total_seconds() >= POLL_SECONDS - TICK_SECONDS / 2 -async def poll(app: "Quart", transport=None) -> None: +async def update_failed(session) -> bool: + """Whether the newest update failed, or the schedule couldn't start one.""" + stopped = (await session.execute(select(HomeAssistantSchedule.stopped))).scalars().first() + newest = (await session.execute(select(HomeAssistantUpdate.state).order_by( + HomeAssistantUpdate.started_at.desc()).limit(1))).scalars().first() + return bool(stopped) or newest == "failed" + + +async def poll(app: "Quart", transport=None, connect=None) -> None: from steward.core.alerts import MetricBatch now = datetime.now(timezone.utc) @@ -64,12 +72,17 @@ async def poll(app: "Quart", transport=None) -> None: connection = (await session.execute(select(HomeAssistantConnection))).scalars().first() if connection is None or not is_due(connection, now): return - snapshot, error, refused = None, None, False + snapshot, health, error, refused = None, None, None, False try: token = await client.access_token(connection, transport) snapshot = read_snapshot(await client.read_states(connection.url, token, connection.verify_tls, transport)) + try: + health = read_health(await client.read_system(connection.url, token, connection.verify_tls, connect)) + except client.Unreachable as exc: + # Updates and backups were read, so it's answering. The last report stands, and says it's old. + health = {**(connection.health or {}), "error": str(exc)} except client.Refused as exc: - error, refused = str(exc), True + snapshot, error, refused = None, str(exc), True except client.Unreachable as exc: error = str(exc) async with app.db_sessionmaker() as session: @@ -83,7 +96,7 @@ async def poll(app: "Quart", transport=None) -> None: batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=0.0, recorded_at=now) else: row.last_ok_at = now - row.updates, row.backup = snapshot["updates"], snapshot["backup"] + row.updates, row.backup, row.health = snapshot["updates"], snapshot["backup"], health batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="is_up", value=1.0, recorded_at=now) batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="updates_waiting", value=len(installable(row)), recorded_at=now) @@ -91,4 +104,13 @@ async def poll(app: "Quart", transport=None) -> None: if age is not None: batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="backup_age_hours", value=age, recorded_at=now) + if not health.get("error"): + if health["repairs_read"]: + batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="problems_open", + value=len(health["problems"]), recorded_at=now) + if health["disk_free_pct"] is not None: + batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="disk_free_pct", + value=health["disk_free_pct"], recorded_at=now) + batch.add(source_module=SOURCE, resource_name=RESOURCE, metric_name="update_failed", + value=1.0 if await update_failed(session) else 0.0, recorded_at=now) await batch.record(session) diff --git a/plugins/homeassistant/service.py b/plugins/homeassistant/service.py index de65328..2afbf2e 100644 --- a/plugins/homeassistant/service.py +++ b/plugins/homeassistant/service.py @@ -1,6 +1,6 @@ # plugins/homeassistant/service.py """Reading what Home Assistant reports: which updates it knows of, which of them -Steward may install, and how its backups stand.""" +Steward may install, how its backups stand, and whether the system has a problem.""" from __future__ import annotations from datetime import datetime, timezone @@ -9,6 +9,7 @@ SOURCE = "homeassistant" RESOURCE = "homeassistant" # Backups run daily, so past a day and a bit one has been missed. STALE_BACKUP_HOURS = 26 +LOW_DISK_PCT = 10 CORE = "update.home_assistant_core_update" OS = "update.home_assistant_operating_system_update" @@ -108,3 +109,50 @@ def backup_verdict(backup: dict | None, now: datetime) -> tuple[str, str]: if age > STALE_BACKUP_HOURS: return "warn", f"The last good backup is {age:.0f} hours old, past the {STALE_BACKUP_HOURS} hours expected." return "ok", "Backups are up to date." + + +def _issue_title(issue: dict, titles: dict) -> str: + key = issue.get("translation_key") or issue.get("issue_id") or "" + title = titles.get(f"component.{issue.get('domain')}.issues.{key}.title") + if not title: + return key.removeprefix("issue_").replace("_", " ").capitalize() or "A repair is open" + for name, value in (issue.get("translation_placeholders") or {}).items(): + title = title.replace("{" + name + "}", str(value)) + return title + + +def read_health(found: dict) -> dict: + """What a poll keeps of client.read_system: the problems to show, and the disk. + A part Home Assistant wouldn't answer is left out rather than read as fine: + `repairs_read` says whether the repairs were seen, and `supervised` whether a + Supervisor answered, as it doesn't on every install or for every user.""" + problems = [] + if (found.get("config") or {}).get("recovery_mode"): + problems.append({"kind": "recovery", "severity": "critical", + "title": "Home Assistant started in recovery mode: its configuration didn't load."}) + issues = found.get("issues") + for issue in issues or []: + if not issue.get("ignored") and not issue.get("dismissed_version"): + problems.append({"kind": "repair", "severity": issue.get("severity") or "warning", + "title": _issue_title(issue, found.get("titles") or {})}) + resolution = found.get("resolution") or {} + for word in ("unhealthy", "unsupported"): + for reason in resolution.get(word) or []: + problems.append({"kind": "supervisor", "severity": "error" if word == "unhealthy" else "warning", + "title": f"The Supervisor calls this system {word}: {str(reason).replace('_', ' ')}."}) + host = found.get("host") or {} + free, total = host.get("disk_free"), host.get("disk_total") + free_pct = round(100 * free / total, 1) if free is not None and total else None + if free_pct is not None and free_pct < LOW_DISK_PCT: + problems.append({"kind": "disk", "severity": "error", "title": f"Only {free:g} GB of {total:g} GB free."}) + boots = found.get("boots") or {} + for addon in found.get("addons") or []: + if boots.get(addon.get("slug")) == "auto": + problems.append({"kind": "addon", "severity": "error", + "title": f"{addon.get('name') or addon.get('slug')} should start on its own and isn't running."}) + return {"repairs_read": issues is not None, "supervised": found.get("resolution") is not None, "problems": problems, + "disk_free_gb": free, "disk_total_gb": total, "disk_free_pct": free_pct} + + +def problems_open(health: dict | None) -> list[dict]: + return (health or {}).get("problems") or [] diff --git a/plugins/homeassistant/templates/homeassistant/form.html b/plugins/homeassistant/templates/homeassistant/form.html index 423f2e1..f0e59c1 100644 --- a/plugins/homeassistant/templates/homeassistant/form.html +++ b/plugins/homeassistant/templates/homeassistant/form.html @@ -23,7 +23,7 @@ -

Steward stores a credential with the same access as the user who signs in. Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status and metrics, take backups and install updates.

+

Steward stores a credential with the same access as the user who signs in. Home Assistant has no narrower permission, so an administrator's credential can do anything that administrator can, including controlling every device. Steward only uses it to read status, take backups and install updates.

We recommend a service account: a Home Assistant user made just for Steward (Settings → People → Add person). What Steward does then appears under its own name in Home Assistant's logs, so it's easy to tell apart from what people do.

Installing updates needs an administrator. With any other user, Steward shows status only.

@@ -38,7 +38,7 @@

Remove this connection

-

Steward stops reading Home Assistant, its metrics included, forgets its credential and ends the sign-in there.

+

Steward stops reading Home Assistant, forgets its credential and ends the sign-in there.

{{ csrf_field() }} diff --git a/plugins/homeassistant/templates/homeassistant/index.html b/plugins/homeassistant/templates/homeassistant/index.html index cdbad8a..962ec67 100644 --- a/plugins/homeassistant/templates/homeassistant/index.html +++ b/plugins/homeassistant/templates/homeassistant/index.html @@ -36,12 +36,33 @@ {% endif %}
+ {{ stat_tile("System problems", problems|length if health.repairs_read else "—", state="degraded" if problems else none) }} + {{ stat_tile("Disk free", (health.disk_free_gb ~ " GB") if health.disk_free_gb is not none and health.disk_free_gb is defined else "—") }} {{ stat_tile("Updates waiting", installable|length if connection.last_ok_at else "—") }} {{ stat_tile("Last good backup", (backup_at|ago) if backup_at else "—", state="degraded" if backup_tone == 'warn' else none) }} {{ stat_tile("Next backup", (backup_next|ago) if backup_next else "—") }}

{{ backup_says }}

+
+

System

+ {% if problems %} + + + + {% for p in problems %}{% endfor %} + +
ProblemSeverity
{{ p.title }}{{ status_badge('down' if p.severity in ('critical', 'error') else 'degraded', p.severity|capitalize) }}
+

Fix these in Home Assistant: Settings → System → Repairs.

+ {% elif health.repairs_read %} +

No repairs open, and nothing wrong reported.

+ {% elif connection.last_ok_at %} +

Home Assistant wouldn't share its repairs with this user. {% if admin %}Sign in again as an administrator to see them.{% endif %}

+ {% endif %} + {% if health.repairs_read and not health.supervised %}

No Supervisor answered, so disk space and add-ons aren't checked here.

{% endif %} + {% if health.error %}

This is from an earlier read. {{ health.error }}

{% endif %} +
+

Updates

{% if installable %} @@ -73,21 +94,36 @@ {% if others %}

Waiting, for Home Assistant to install: {% for u in others %}{{ u.title }}{{ ", " if not loop.last }}{% endfor %}. Steward installs Core, OS and add-on updates only.

{% endif %}
-{% if metrics %} -
-

Metrics

- {% if metrics.uses_sign_in %} -

{{ status_badge(metrics.reach) }} Read with this sign-in into the target {{ metrics.name }}: {{ metrics.says }}{% if metrics.reach == 'up' %} {{ metrics.kept }} series kept.{% endif %}

- {% elif metrics.id %} -

The target {{ metrics.name }} reads these with a token of its own: {{ metrics.says }}

- {% if admin %}

Use this sign-in instead, so one Home Assistant user covers everything. The target keeps its name, filters and history, and forgets its token.

- {{ csrf_field() }}{% endif %} +
+

Scheduled updates

+ {% if plan and plan.enabled %} +

{{ plan_when }}. Next: {{ plan_next|ago }}. Add-ons first, then Core, then the OS, each after a confirmed backup. A failure ends the run and the rest wait.

{% else %} -

Home Assistant's entity readings can be kept as metrics too, for charts and alert rules, with this sign-in and nothing else to set up.

- {% if admin %}
{{ csrf_field() }}
{% endif %} +

Off. Updates are only installed when someone chooses Install.

+ {% endif %} + {% if plan and plan.stopped %}

The last run stopped early. {{ plan.stopped }}

{% endif %} + {% if admin %} +

A Core update restarts Home Assistant and an OS update reboots the machine, with nobody there to watch. An update can also change how automations behave.

+
{{ csrf_field() }} + +
+ + +
+
+ + +
+
+ + +
+ +
{% endif %}
-{% endif %} {% if runs %}
diff --git a/plugins/host_agent/cadence.py b/plugins/host_agent/cadence.py index 2101709..9741fcd 100644 --- a/plugins/host_agent/cadence.py +++ b/plugins/host_agent/cadence.py @@ -21,11 +21,12 @@ the host it runs on, picks up where it was. from __future__ import annotations import json -from datetime import datetime, time, timedelta, timezone +from datetime import datetime, timedelta + +from steward.core.weekly import WEEKDAYS, describe, fired_slot, is_due, last_slot, next_slot, parse_at # noqa: F401 from . import maintenance, rebalance, timeline -WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday") # How long a host may take to be back in service after its job finished. SETTLE_LIMIT = timedelta(minutes=30) # An agent heard from this recently is posting. @@ -33,58 +34,6 @@ POSTING_WITHIN = timedelta(minutes=3) RUN_STATES = ("running", "done", "failed", "stopped") -def _zone(name: str): - from steward.core.formatting import zone - return zone(name) - - -def parse_at(at_time: str) -> time: - h, m = (int(x) for x in at_time.split(":", 1)) - if not (0 <= h < 24 and 0 <= m < 60): - raise ValueError(at_time) - return time(h, m) - - -def last_slot(plan, now: datetime) -> datetime: - """The most recent scheduled time at or before `now`, in UTC.""" - tz = _zone(plan.time_zone) - local = now.astimezone(tz) - day = local.date() - timedelta(days=(local.weekday() - plan.weekday) % 7) - slot = datetime.combine(day, parse_at(plan.at_time), tzinfo=tz) - if slot > local: - slot -= timedelta(days=7) - return slot.astimezone(timezone.utc) - - -def fired_slot(plan) -> datetime | None: - """The last slot that fired, or None while none has. A new plan's - last_slot_at holds the moment it was made, so its first slot is the - next one rather than one already past; that moment is no slot (#5863).""" - at = plan.last_slot_at - if at is None: - return None - return at if last_slot(plan, at) == at else None - - -def next_slot(plan, now: datetime) -> datetime: - return last_slot(plan, now) + timedelta(days=7) - - -def is_due(plan, now: datetime) -> datetime | None: - """The slot to fire now, or None. A slot fires once (last_slot_at), and - only while its window is still open: a Steward that was down over the slot - starts late, not a day late.""" - if not plan.enabled: - return None - slot = last_slot(plan, now) - since = plan.last_slot_at or plan.created_at - if since is not None and slot <= since: - return None - if now - slot >= timedelta(minutes=plan.window_minutes): - return None - return slot - - def back_in_service(*, last_seen_at, node, services, job_finished_at, now: datetime, expect_active: bool = True) -> str | None: """None when the host is back in service, else why not yet. `node` is the @@ -134,13 +83,6 @@ def next_move(run, plan, job, back: str | None, now: datetime) -> tuple[str, str return "settling", back -def describe(plan) -> str: - """"Sundays at 03:00 (Europe/London), up to 4 h" for the page.""" - hours = plan.window_minutes / 60 - span = f"{hours:g} h" if plan.window_minutes % 60 == 0 else f"{plan.window_minutes} min" - return f"{WEEKDAYS[plan.weekday]}s at {plan.at_time} ({plan.time_zone}), up to {span}" - - # ── applied to the database: the minute tick, Run now, the page ────────────── async def group_hosts(session, group_id: str) -> list: diff --git a/plugins/prometheus/migrations/versions/prometheus_005_own_tokens.py b/plugins/prometheus/migrations/versions/prometheus_005_own_tokens.py new file mode 100644 index 0000000..1e62cb0 --- /dev/null +++ b/plugins/prometheus/migrations/versions/prometheus_005_own_tokens.py @@ -0,0 +1,30 @@ +"""Every target holds its own token: one that took it from a sign-in goes + +Revision ID: prometheus_005_own_tokens +Revises: prometheus_004_token_from +Create Date: 2026-11-01 +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "prometheus_005_own_tokens" +down_revision: Union[str, None] = "prometheus_004_token_from" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Left in place, such a target would ask with no token, and some services ban + # an address for that. Its newest values go with it; history ages out. + op.execute(""" + DELETE FROM plugin_metric_latest l USING prometheus_targets t + WHERE t.token_from IS NOT NULL AND l.source_module = 'prometheus' + AND (l.resource_name = t.name OR left(l.resource_name, length(t.name) + 1) = t.name || ':')""") + op.execute("DELETE FROM prometheus_targets WHERE token_from IS NOT NULL") + op.drop_column("prometheus_targets", "token_from") + + +def downgrade() -> None: + op.add_column("prometheus_targets", sa.Column("token_from", sa.String(50), nullable=True)) diff --git a/plugins/prometheus/models.py b/plugins/prometheus/models.py index 2e63a9c..f723854 100644 --- a/plugins/prometheus/models.py +++ b/plugins/prometheus/models.py @@ -13,8 +13,7 @@ from steward.models.base import Base class PrometheusTarget(Base): """A /metrics endpoint Steward scrapes, edited on the Prometheus page. - The bearer token is stored encrypted, or `token_from` names the plugin whose - sign-in supplies it at each scrape (its metrics.scrape_token). keep and drop are lists of name + The bearer token is stored encrypted. keep and drop are lists of name patterns ("fabledarchives_*") an admin chooses: Steward does the filtering, the endpoint serves everything. The last_* columns are the newest scrape's outcome, written by the scraper, so a page can say why a @@ -27,7 +26,6 @@ class PrometheusTarget(Base): name: Mapped[str] = mapped_column(String(100), nullable=False, unique=True) url: Mapped[str] = mapped_column(String(500), nullable=False) token: Mapped[str] = mapped_column(Text, nullable=False, default="") - token_from: Mapped[str | None] = mapped_column(String(50), nullable=True) verify_tls: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) interval_seconds: Mapped[int] = mapped_column(Integer, nullable=False, default=60) keep: Mapped[list] = mapped_column(JSON, nullable=False, default=list) diff --git a/plugins/prometheus/routes.py b/plugins/prometheus/routes.py index 082fb19..01ad8dc 100644 --- a/plugins/prometheus/routes.py +++ b/plugins/prometheus/routes.py @@ -12,14 +12,13 @@ from sqlalchemy import delete, func, select, update from steward.auth.middleware import require_role from steward.core.audit import log_audit +from steward.core.crypto import decrypt_secret from steward.core.metric_series import latest_values, points -from steward.core.prometheus import TokenUnavailable from steward.models.metrics import PluginMetric, PluginMetricLatest from steward.models.users import UserRole from . import library from .models import PrometheusTarget, PrometheusWidgetTemplate -from .scheduler import target_token from .scraper import ScrapeError, scrape from .service import (OWN_METRICS, SOURCE, FormError, fold_states, form_values, is_moment, map_families, parse_form, patterns, shown_value, target_state) @@ -81,8 +80,7 @@ def _typed(form, target: PrometheusTarget | None = None) -> dict: return {"name": form.get("name", ""), "url": form.get("url", ""), "interval_seconds": form.get("interval_seconds", ""), "series_cap": form.get("series_cap", ""), "keep": form.get("keep", ""), "drop": form.get("drop", ""), - "verify_tls": bool(form.get("verify_tls")), "has_token": bool(target and target.token), - "token_from": target.token_from if target else None} + "verify_tls": bool(form.get("verify_tls")), "has_token": bool(target and target.token)} async def _name_taken(db, name: str, other_than: str | None = None) -> bool: @@ -237,15 +235,10 @@ async def test_target(): existing = await db.get(PrometheusTarget, form["target_id"]) url = (form.get("url") or "").strip() token = (form.get("token") or "").strip() + if not token and existing is not None and existing.token and not form.get("clear_token"): + token = decrypt_secret(existing.token, context=f"prometheus target {existing.name} token") found: dict = {} - if not token and existing is not None and not form.get("clear_token"): - try: - token = await target_token(current_app, existing, url) - except TokenUnavailable as exc: - found["error"] = str(exc) - if found: - pass - elif not url.startswith(("http://", "https://")): + if not url.startswith(("http://", "https://")): found["error"] = "Enter the address first: http://… or https://…" else: try: diff --git a/plugins/prometheus/scheduler.py b/plugins/prometheus/scheduler.py index 638cf02..dba104c 100644 --- a/plugins/prometheus/scheduler.py +++ b/plugins/prometheus/scheduler.py @@ -48,33 +48,14 @@ def is_due(target, now: datetime) -> bool: return elapsed >= target.interval_seconds - TICK_SECONDS / 2 -def _token_source(name: str): - from steward.core.plugin_manager import loaded_plugin_submodule - return loaded_plugin_submodule(name, "metrics") - - -async def target_token(app: "Quart", target, url: str) -> str: - """The token to ask `url` with: the target's own, or one from the plugin whose - sign-in it uses. Raises TokenUnavailable when that plugin has none to give.""" - from steward.core.crypto import decrypt_secret - from steward.core.prometheus import TokenUnavailable - - if not target.token_from: - return decrypt_secret(target.token, context=f"prometheus target {target.name} token") - source = _token_source(target.token_from) - if source is None: - raise TokenUnavailable(f"Its token comes from the {target.token_from} plugin, which isn't turned on.") - return await source.scrape_token(app, url) - - async def scrape_due(app: "Quart", scrape=None) -> None: """Scrape every due target at once, then write each one's outcome to its row and its series to the metric tables. `scrape` stands in for scraper.scrape in tests.""" - from steward.core.prometheus import TokenUnavailable + from steward.core.crypto import decrypt_secret from .models import PrometheusTarget - from .scraper import ScrapeError, TokenRefused + from .scraper import ScrapeError from .scraper import scrape as fetch scrape = scrape or fetch @@ -87,15 +68,11 @@ async def scrape_due(app: "Quart", scrape=None) -> None: async def one(t): started = time.monotonic() - - async def fetched(): - return await scrape(t.url, await target_token(app, t, t.url), t.verify_tls) - + token = decrypt_secret(t.token, context=f"prometheus target {t.name} token") try: - families = await asyncio.wait_for(fetched(), SCRAPE_DEADLINE_SECONDS) + families = await asyncio.wait_for(scrape(t.url, token, t.verify_tls), + SCRAPE_DEADLINE_SECONDS) return families, (time.monotonic() - started) * 1000 - except TokenUnavailable as exc: - return ScrapeError(str(exc)), (time.monotonic() - started) * 1000 except TimeoutError: return ScrapeError(f"Gave up after {SCRAPE_DEADLINE_SECONDS} s."), \ (time.monotonic() - started) * 1000 @@ -104,11 +81,6 @@ async def scrape_due(app: "Quart", scrape=None) -> None: results = await asyncio.gather(*(one(t) for t in targets)) await record(app, targets, results, datetime.now(timezone.utc)) - # A refused sign-in is the lending plugin's to stop, so one Try again or sign-in - # there starts everything that uses it. - for t, (result, _) in zip(targets, results): - if t.token_from and isinstance(result, TokenRefused) and (source := _token_source(t.token_from)): - await source.token_refused(app) async def forget_gone(session, target: str, kept: set[tuple[str, str]]) -> None: @@ -160,7 +132,7 @@ async def record(app: "Quart", targets, results, now: datetime) -> None: if not isinstance(result, ScrapeError): logger.warning("Prometheus scrape of %s failed: %r", target.name, result) row.last_error = str(result) or "The scrape failed." - row.refused = isinstance(result, TokenRefused) and not row.token_from + row.refused = isinstance(result, TokenRefused) batch.add(source_module=SOURCE, resource_name=row.name, metric_name=IS_UP, value=0.0, recorded_at=now) continue diff --git a/plugins/prometheus/scraper.py b/plugins/prometheus/scraper.py index f59a36b..9c015b8 100644 --- a/plugins/prometheus/scraper.py +++ b/plugins/prometheus/scraper.py @@ -23,10 +23,6 @@ class TokenRefused(ScrapeError): Home Assistant, for one, bans an address after a few refusals.""" -class NoMetricsHere(ScrapeError): - """The address answered 404: nothing is serving metrics there.""" - - async def scrape(url: str, token: str = "", verify_tls: bool = True, transport: httpx.AsyncBaseTransport | None = None) -> dict[str, prometheus.Family]: """The endpoint's families. `transport` stands in for the network in tests.""" @@ -40,8 +36,6 @@ async def scrape(url: str, token: str = "", verify_tls: bool = True, if response.status_code in (401, 403): raise TokenRefused(f"Answered {response.status_code}: the token was refused. " "Check the token, or make a new one on the service.") - if response.status_code == 404: - raise NoMetricsHere(f"Answered 404 {response.reason_phrase}, not metrics.") if response.status_code != 200: raise ScrapeError(f"Answered {response.status_code} {response.reason_phrase}, not metrics.") body = bytearray() diff --git a/plugins/prometheus/service.py b/plugins/prometheus/service.py index dafa113..6345083 100644 --- a/plugins/prometheus/service.py +++ b/plugins/prometheus/service.py @@ -212,7 +212,7 @@ def form_values(target: PrometheusTarget | None) -> dict: return {"name": target.name, "url": target.url, "interval_seconds": target.interval_seconds, "series_cap": target.series_cap, "verify_tls": target.verify_tls, "keep": "\n".join(target.keep or []), - "drop": "\n".join(target.drop or []), "has_token": bool(target.token), "token_from": target.token_from} + "drop": "\n".join(target.drop or []), "has_token": bool(target.token)} def _bounded(form, key: str, default: int, low: int, high: int, says: str) -> int: @@ -250,7 +250,6 @@ def parse_form(form, existing: PrometheusTarget | None = None) -> dict: else: token = "" return {"name": name, "url": url, "token": token, - "token_from": None if typed or existing is None else existing.token_from, "verify_tls": bool(form.get("verify_tls")), "interval_seconds": interval, "keep": patterns(form.get("keep") or ""), "drop": patterns(form.get("drop") or ""), "series_cap": cap} diff --git a/plugins/prometheus/templates/prometheus/form.html b/plugins/prometheus/templates/prometheus/form.html index 201718b..586c32a 100644 --- a/plugins/prometheus/templates/prometheus/form.html +++ b/plugins/prometheus/templates/prometheus/form.html @@ -37,12 +37,8 @@
- {% if v.token_from %} - Comes from the sign-in on the {{ v.token_from }} plugin's page, so that one user covers this too. Typing a token here uses that instead. - {% else %} + placeholder="{{ 'kept; type to change' if v.has_token else 'none' }}"> Sent as "Authorization: Bearer …". Blank {{ "keeps the saved one" if v.has_token else "sends none" }}. - {% endif %} {% if v.has_token %}