Compare commits
1
Commits
1d9ac2b46a
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 bvandeusen
|
001cbafe56 |
@@ -14,17 +14,6 @@ on:
|
||||
# pull_request intentionally absent — push on [dev, main] already fires CI for
|
||||
# every dev commit and dev→main PRs. Single-operator repo, no fork PRs.
|
||||
|
||||
# Serialize CI runs per ref so the publish lane never shares the runner's docker
|
||||
# daemon with another run. Two publishes racing on one daemon evict each other's
|
||||
# freshly-built image mid-push, breaking the post-build tag/push (issue #1093).
|
||||
# cancel-in-progress:false is deliberate — rule 46 requires EVERY dev/main push to
|
||||
# publish its own immutable :<sha> image, so a superseded run must still run to
|
||||
# completion (never cancelled) to emit its SHA. (Forgejo honors `concurrency:` —
|
||||
# CI-runner's build workflows use it.)
|
||||
concurrency:
|
||||
group: ci-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so this
|
||||
# runs with NO dependency install and surfaces lint bounces in seconds.
|
||||
@@ -126,25 +115,14 @@ jobs:
|
||||
run: |
|
||||
set -euxo pipefail
|
||||
IMAGE=git.fabledsword.com/bvandeusen/steward
|
||||
# The moving tag for this ref: dev→:dev, main→:latest (rule 46). main IS
|
||||
# the production line, so :latest tracks main's tip; there is no :main.
|
||||
MOVING=""
|
||||
docker build -t "$IMAGE:${{ github.sha }}" .
|
||||
docker push "$IMAGE:${{ github.sha }}"
|
||||
if [ "${{ github.ref }}" = "refs/heads/dev" ]; then
|
||||
MOVING="dev"
|
||||
docker tag "$IMAGE:${{ github.sha }}" "$IMAGE:dev"
|
||||
docker push "$IMAGE:dev"
|
||||
elif [ "${{ github.ref }}" = "refs/heads/main" ]; then
|
||||
MOVING="latest"
|
||||
fi
|
||||
# Tag BOTH targets at build time (mirrors CI-runner's build workflows):
|
||||
# one `docker build -t :<sha> -t :<moving>` points both tags at the image
|
||||
# atomically, so we never run a post-push `docker tag` of an image a
|
||||
# concurrent run could have evicted from the shared daemon (issue #1093).
|
||||
if [ -n "$MOVING" ]; then
|
||||
docker build -t "$IMAGE:${{ github.sha }}" -t "$IMAGE:$MOVING" .
|
||||
docker push "$IMAGE:${{ github.sha }}"
|
||||
docker push "$IMAGE:$MOVING"
|
||||
else
|
||||
docker build -t "$IMAGE:${{ github.sha }}" .
|
||||
docker push "$IMAGE:${{ github.sha }}"
|
||||
docker tag "$IMAGE:${{ github.sha }}" "$IMAGE:latest"
|
||||
docker push "$IMAGE:latest"
|
||||
fi
|
||||
- name: Prune dangling layers
|
||||
if: always()
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||
|
||||
- Steward has **no frontend and no Redis/Celery** (no external workers), so there
|
||||
is no frontend-build lane and the only service container is Postgres.
|
||||
- Integration uses Fabled-Git Actions `services:` + a socket-discovered bridge IP
|
||||
- Integration uses Forgejo Actions `services:` + a socket-discovered bridge IP
|
||||
because `act_runner` (swarm-runner v0.6+) puts services on the default bridge
|
||||
with no embedded DNS. FabledCurator's `ci.yml` is the canonical example of the
|
||||
pattern; Steward's is the same shape minus Redis and minus sharding.
|
||||
|
||||
@@ -13,14 +13,6 @@
|
||||
database:
|
||||
url: "postgresql+asyncpg://steward:password@localhost/steward"
|
||||
|
||||
# Optional: seconds to wait at startup for the database to become reachable
|
||||
# before giving up (default: 60). Steward retries with backoff rather than
|
||||
# crashing when the DB is merely slow to arrive — replaying WAL after an
|
||||
# unclean shutdown, or container DNS not yet resolving after a host reboot.
|
||||
# Raise it if your database is routinely slower than this to accept
|
||||
# connections. Env var: STEWARD_DB_CONNECT_TIMEOUT
|
||||
# connect_timeout: 60
|
||||
|
||||
# Optional: override the auto-generated secret key.
|
||||
# If not set, a key is auto-generated on first run and saved to /data/secret.key.
|
||||
# secret_key: "change-me-to-a-random-string"
|
||||
|
||||
@@ -44,30 +44,6 @@ All routes are Quart Blueprints registered in `app.py`:
|
||||
|
||||
Plugin blueprints are mounted automatically by `load_plugins()` using the plugin directory name as the URL prefix.
|
||||
|
||||
`/ai-access/` (`ai_access_bp`, `steward/ai_access/routes.py`) is where a signed-in user manages their API tokens.
|
||||
|
||||
### `/mcp`: outside the blueprint pipeline
|
||||
|
||||
`mount_mcp(app)` (`steward/mcp/server.py`) wraps `app.asgi_app`. A request to
|
||||
`/mcp` is handed to the MCP SDK's streamable-HTTP app before Quart sees it, so
|
||||
no blueprint, session cookie or `before_request` hook runs for it. Each
|
||||
request then goes through these steps:
|
||||
|
||||
1. **Database gate.** If the database is unreachable, the request gets a 503
|
||||
with a readable message.
|
||||
2. **Bearer resolution.** `api_tokens.resolve_token()` turns the token into a
|
||||
`Caller` (user, effective role, token id). A missing, revoked or expired
|
||||
token gets a 401.
|
||||
3. **Caller context.** The `Caller` and the app are set in ContextVars
|
||||
(`steward/mcp/_context.py`). Tools read them through `current_caller()` and
|
||||
`mcp_app()`, because Quart's `current_app` and `session` do not exist there.
|
||||
4. **Role gate.** `StewardMCPServer.refusal()` hides and refuses tools above
|
||||
the caller's role, and plugin tools whose plugin is disabled.
|
||||
|
||||
The SDK's session manager runs in Quart's `before_serving`/`after_serving`
|
||||
lifecycle. The server is stateless: no MCP session outlives a request. See
|
||||
[MCP server](core/mcp.md) for the security model and the tools.
|
||||
|
||||
---
|
||||
|
||||
## Scheduler
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
# MCP server
|
||||
|
||||
Steward serves the [Model Context Protocol](https://modelcontextprotocol.io) at
|
||||
`/mcp`. An AI client connected to it can look into everything Steward watches
|
||||
(hosts, monitors, alerts, metrics, containers, playbook runs) and, with a
|
||||
high enough role, change it. A token's role sets that ceiling.
|
||||
|
||||
**Source:** `steward/mcp/`. The tools live in `steward/mcp/tools/`, and a
|
||||
plugin's tools live in `plugins/<name>/mcp_tools.py`.
|
||||
**Transport:** streamable HTTP, stateless. Every request carries its own
|
||||
token, so a client survives a Steward restart without reconnecting.
|
||||
|
||||
---
|
||||
|
||||
## Connecting
|
||||
|
||||
1. In Steward, open **AI access** and create a token. Pick the lowest role
|
||||
that does the job.
|
||||
2. Copy the token (it is shown once) and use one of the recipes on that
|
||||
page. They carry the real endpoint URL and the new token. For Claude Code:
|
||||
|
||||
```sh
|
||||
claude mcp add --transport http steward https://steward.example/mcp \
|
||||
--header "Authorization: Bearer stw_…"
|
||||
```
|
||||
|
||||
3. Ask the client to call `whoami`. It answers with the user, the effective
|
||||
role and what that role allows.
|
||||
|
||||
Claude Desktop connects through `mcp-remote`. Open WebUI and other clients
|
||||
need an MCP server with the **Streamable HTTP** transport and bearer
|
||||
authentication. The AI access page shows the recipes for both.
|
||||
|
||||
### Behind a reverse proxy
|
||||
|
||||
- The URL on the AI access page is **Settings → General → Public base URL**
|
||||
plus `/mcp`. If that setting is empty, the URL is built from the request's
|
||||
host instead.
|
||||
- The proxy must pass the `Authorization` header through unchanged.
|
||||
- It must not buffer responses. Replies may arrive as server-sent events
|
||||
(`text/event-stream`). For nginx, set `proxy_buffering off;` on the
|
||||
location.
|
||||
- Steward does no Host-header (DNS-rebinding) check on `/mcp`, because the
|
||||
bearer token is the security boundary. Serve it over HTTPS.
|
||||
|
||||
---
|
||||
|
||||
## Security model
|
||||
|
||||
| Rule | How it is enforced |
|
||||
|---|---|
|
||||
| Bearer tokens only, no OAuth | `steward/core/api_tokens.py`. Only a hash is stored, and the token is shown once. |
|
||||
| A token never acts above its user | Effective role = min(token role, user's **current** role). Lowering a user lowers every token they hold; disabling the user stops them all. |
|
||||
| Default deny | A tool is reachable only if it was registered with a role (`register_tool(fn, role=..., read_only=...)`). `tools/list` shows a caller only the tools their role allows. A call above the role is refused, and the refusal names the role it needs. |
|
||||
| Plugin tools follow the plugin | A disabled plugin's tools are hidden and refused. |
|
||||
| Dry run first | `run_playbook`, `create_schedule` and `set_alert_action` default to Ansible check mode (`--check --diff`). Changes are applied only when the caller passes `check=false`. A playbook its author flagged `# steward:confirm: true` also needs `confirmed=true`. `prune_docker` previews unless `check=false`. |
|
||||
| Secrets stay put | Playbook variables Steward detects as secret reach the run but are never stored, audited or returned. Schedules and alert actions refuse them, since they would have to store them. Inventory variables are listed by name only, and HTTP monitor header values are redacted. |
|
||||
| Every change is audited | Each write records an audit event as `<user> via MCP`, with the token id in its detail. `list_audit_events(user="<user>")` includes those events. |
|
||||
| Unknown arguments are refused | A misspelt argument is rejected with a "did you mean", rather than silently dropped. |
|
||||
|
||||
Tokens never mint or revoke tokens; that happens only on the AI access
|
||||
page, in a browser session.
|
||||
|
||||
---
|
||||
|
||||
## Tools by role
|
||||
|
||||
Each role includes the tools of the roles above it in this list.
|
||||
|
||||
### Viewer: read everything
|
||||
|
||||
| Area | Tools |
|
||||
|---|---|
|
||||
| Start here | `whoami`, `get_overview` (what is wrong now, ranked, each finding with the call that shows its evidence) |
|
||||
| Hosts and metrics | `list_hosts`, `get_host`, `get_host_metrics`, `query_metrics` |
|
||||
| Monitors | `list_monitors`, `get_monitor` |
|
||||
| Alerts | `list_alerts`, `get_alert_history`, `list_alert_rules`, `get_alert_rule_options`, `list_maintenance_windows` |
|
||||
| Automation | `list_playbooks`, `get_playbook`, `list_inventory`, `list_ansible_runs`, `get_ansible_run`, `wait_for_run`, `list_schedules` |
|
||||
| Audit | `list_audit_events` |
|
||||
| Docker plugin | `list_containers`, `get_container`, `list_swarm_services`, `search_container_logs`, `get_docker_disk_usage` |
|
||||
| Host agent plugin | `list_agents` |
|
||||
|
||||
### Operator: run playbooks, change monitoring
|
||||
|
||||
| Area | Tools |
|
||||
|---|---|
|
||||
| Automation | `run_playbook`, `cancel_ansible_run`, `create_schedule`, `update_schedule`, `delete_schedule` |
|
||||
| Monitors | `create_monitor`, `update_monitor`, `set_monitor_enabled` |
|
||||
| Alerts | `create_alert_rule`, `update_alert_rule`, `acknowledge_alert`, `create_maintenance_window`, `end_maintenance_window` |
|
||||
|
||||
### Admin: destructive and remediation
|
||||
|
||||
`set_alert_action` (the playbook a rule runs when it fires), `delete_monitor`,
|
||||
`delete_alert_rule`, `prune_docker`.
|
||||
|
||||
The client sees each tool's full contract: arguments, what an empty value
|
||||
means, and what comes back. That contract is the tool's docstring, so read the
|
||||
docstrings in `steward/mcp/tools/` for the detail.
|
||||
|
||||
---
|
||||
|
||||
## Response conventions
|
||||
|
||||
Every tool follows these conventions (`steward/mcp/tools/_shape.py`):
|
||||
|
||||
- **Times** are ISO-8601 UTC with a `Z`. Anything with a "when" also carries
|
||||
`age_s`, so the client never does clock arithmetic.
|
||||
- **Lists are bounded.** A list response carries `total` and `truncated`. When
|
||||
it was cut, `more` names the argument that fetches the rest. Nothing is cut
|
||||
silently.
|
||||
- **Time series** come back as a summary (min/avg/max/p95/last) plus a series
|
||||
of at most 60 points, never raw rows.
|
||||
- **Lookup by name or id.** A miss suggests close matches and names the tool
|
||||
that lists the valid choices.
|
||||
- **Refusals are readable.** Bad input becomes a tool error that says what was
|
||||
wrong, why, and what to do instead, and nothing is changed. The tools share
|
||||
their validation with the web forms (`monitors/service.py`,
|
||||
`alerts/service.py`), so a value the UI refuses is refused here for the same
|
||||
reason.
|
||||
- **Database outages** return a plain "Steward cannot reach its database"
|
||||
rather than a stack trace.
|
||||
|
||||
---
|
||||
|
||||
## Adding tools
|
||||
|
||||
Core tools: add an `async def` to a module in `steward/mcp/tools/` and list it
|
||||
in that module's `register()` with its minimum role and `read_only`. A plugin
|
||||
exposes `register_mcp_tools(register)`, and optionally `mcp_overview(db)` to
|
||||
contribute findings to `get_overview`. See
|
||||
[Writing a plugin](../plugins/writing-a-plugin.md). A tool that changes
|
||||
anything calls `audit_mcp(...)` from `steward/mcp/tools/_audit.py`.
|
||||
@@ -74,15 +74,8 @@ token = a1b2c3d4...
|
||||
interval_seconds = 30
|
||||
hostname = myhost # optional; defaults to uname -n
|
||||
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
|
||||
docker_logs_enabled = true # optional; on by default — per-host container-log kill-switch
|
||||
docker_log_exclude = watchtower, noisy-svc # optional; container names whose logs this host skips collecting
|
||||
```
|
||||
|
||||
Container-log collection is on by default (no config needed). The two keys above
|
||||
are the per-host opt-outs; the operator-facing global toggle + exclude live in
|
||||
Settings and are enforced server-side (the push model has no channel to command
|
||||
an agent), so they don't require touching a host's conf.
|
||||
|
||||
### Agent internals (function list, not classes)
|
||||
|
||||
- `read_config(path)` — parses the conf file into a dict.
|
||||
@@ -92,17 +85,10 @@ an agent), so they don't require touching a host's conf.
|
||||
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
|
||||
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
|
||||
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
|
||||
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
|
||||
container, fetches new log lines over the Docker socket with an incremental
|
||||
`since` cursor kept per container in `state` (a container's first interval
|
||||
seeds from a short tail). Demuxes the multiplexed stream, parses each line's
|
||||
RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
|
||||
a truncation; the deferred lines come next interval). Folded into the sample as
|
||||
`docker_logs`; omitted when empty.
|
||||
- `build_payload()` — assembles a snapshot from all collectors into one dict.
|
||||
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
|
||||
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
|
||||
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).
|
||||
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer.
|
||||
|
||||
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
|
||||
|
||||
@@ -269,7 +255,6 @@ Content-Type: application/json
|
||||
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
|
||||
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
|
||||
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
|
||||
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.
|
||||
|
||||
### Server expansion into `PluginMetric` rows
|
||||
|
||||
|
||||
@@ -323,61 +323,6 @@ Role hierarchy: `admin > operator > viewer`. Requiring `viewer` grants access to
|
||||
|
||||
---
|
||||
|
||||
## Agent Tools (MCP)
|
||||
|
||||
Steward serves an MCP endpoint at `/mcp` that AI clients reach with a personal API token created on the AI access page. A plugin adds tools to it by exporting an optional `register_mcp_tools(register)` from `__init__.py`:
|
||||
|
||||
```python
|
||||
# plugins/myplugin/__init__.py
|
||||
def register_mcp_tools(register) -> None:
|
||||
from .mcp_tools import list_targets, rescan_target
|
||||
from steward.models.users import UserRole
|
||||
register(list_targets, role=UserRole.viewer, read_only=True)
|
||||
register(rescan_target, role=UserRole.operator, read_only=False)
|
||||
```
|
||||
|
||||
```python
|
||||
# plugins/myplugin/mcp_tools.py
|
||||
from steward.mcp._context import current_caller, mcp_app
|
||||
|
||||
|
||||
async def list_targets(limit: int = 50) -> dict:
|
||||
"""What this tool is for, when to use it over its siblings, what each
|
||||
argument's empty value means, and what comes back. This docstring is the
|
||||
tool's description: it is all the agent knows about it."""
|
||||
app = mcp_app() # Quart's current_app does not exist here
|
||||
async with app.db_sessionmaker() as db:
|
||||
...
|
||||
return {"targets": [...], "total": n, "truncated": n > limit}
|
||||
```
|
||||
|
||||
- `role` and `read_only` are required. A tool is listed and callable only for callers whose role reaches `role`, and only while the plugin is enabled.
|
||||
- Tools run outside Quart's request pipeline: use `mcp_app()` and `current_caller()` from `steward.mcp._context`, never `current_app` or `session`.
|
||||
- Refuse by raising `ValueError("what was wrong, and what to do instead")`. The agent reads that text. Any other exception reaches it only as a generic failure.
|
||||
- Tool names are global across core and every plugin; a name already taken fails registration and the plugin contributes no tools (logged).
|
||||
- Hot-reloading the plugin replaces its tools.
|
||||
- Follow the response conventions the core tools use (`steward/mcp/tools/_shape.py`): times via `iso()` plus `age_s()`, lists through `bounded()` so a cut is never silent, series through `summarize()`, and lookups by name or id through `pick()`.
|
||||
|
||||
### Contributing to get_overview
|
||||
|
||||
`get_overview` is the agent's first call: what is wrong right now, ranked. A plugin adds its own problems to it by exporting an optional `mcp_overview(db)` that returns findings built with `steward.mcp.tools.overview.finding`:
|
||||
|
||||
```python
|
||||
# plugins/myplugin/__init__.py
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
from steward.mcp.tools.overview import finding
|
||||
return [
|
||||
finding("warning", "target_unreachable", t.name, f"{t.name} has not answered for {t.misses} scans",
|
||||
since=t.last_ok_at, ids={"target_id": t.id}, next=f'get_target(target="{t.id}")')
|
||||
for t in await _unreachable_targets(db)
|
||||
]
|
||||
```
|
||||
|
||||
- `severity` is `"critical"` (something is down or about to be) or `"warning"` (degraded, or heading that way). Report problems only; a healthy plugin returns `[]`.
|
||||
- It is called only while the plugin is enabled, with its own session. If it raises, the overview still returns, and names the plugin under `unavailable`.
|
||||
|
||||
---
|
||||
|
||||
## Publishing to the Catalog
|
||||
|
||||
The official plugin catalog is hosted at `https://git.fabledsword.com/bvandeusen/Steward-plugins`. Anyone can submit a plugin by opening a pull request — first-party and third-party plugins are treated identically by the catalog system.
|
||||
@@ -468,7 +413,6 @@ Hot-reload works reliably for brand-new plugin installs. Updates to already-acti
|
||||
- [ ] Revision IDs prefixed with plugin name
|
||||
- [ ] `record_metric()` called inside `session.begin()`
|
||||
- [ ] Routes use `@require_role` decorator
|
||||
- [ ] MCP tools (if any) registered via `register_mcp_tools(register)` with a role and `read_only`
|
||||
- [ ] Templates namespaced under `templates/myplugin/`
|
||||
- [ ] Plugin enabled in app settings
|
||||
- [ ] `index.yaml` entry added with `download_url` and `checksum_sha256`
|
||||
|
||||
@@ -136,25 +136,3 @@ Quick reference for where key functions, models, and entry points live in the co
|
||||
| `steward/migrations/versions/` | Core schema (hosts, users, monitors, alerts, app_settings) |
|
||||
| `plugins/traefik/migrations/versions/` | `traefik_metrics` table |
|
||||
| `alembic.ini` | Alembic config; `version_locations` lists all migration directories |
|
||||
|
||||
---
|
||||
|
||||
## MCP server and API tokens
|
||||
|
||||
| What | File | Function / Class |
|
||||
|---|---|---|
|
||||
| Mount `/mcp` (ASGI dispatch, bearer auth, DB gate) | `steward/mcp/server.py` | `mount_mcp(app)` |
|
||||
| Server, role registry, refusals | `steward/mcp/server.py` | `StewardMCPServer`, `register_tool()`, `refusal()` |
|
||||
| Plugin tools and overview findings | `steward/mcp/server.py` | `register_plugin(name, module)` |
|
||||
| Caller and app inside a tool | `steward/mcp/_context.py` | `current_caller()`, `mcp_app()` |
|
||||
| Core tool modules | `steward/mcp/tools/` | `register_all()`; one module per area |
|
||||
| Response conventions | `steward/mcp/tools/_shape.py` | `iso`, `age_s`, `bounded`, `lookback`, `summarize`, `pick` |
|
||||
| Audit a write as "<user> via MCP" | `steward/mcp/tools/_audit.py` | `audit_mcp()` |
|
||||
| "What is wrong now" | `steward/mcp/tools/overview.py` | `get_overview()` |
|
||||
| Playbooks, runs, schedules, alert actions | `steward/mcp/tools/automation.py` | `run_playbook()`, `wait_for_run()`, … |
|
||||
| Monitor validation (forms and tools) | `steward/monitors/service.py` | `monitor_fields()`, `build_config()` |
|
||||
| Alert rule / window validation | `steward/alerts/service.py` | `rule_fields()`, `window_fields()`, `acknowledge()` |
|
||||
| Docker and host-agent tools | `plugins/docker/mcp_tools.py`, `plugins/host_agent/mcp_tools.py` | `register_tools()`, `mcp_overview()` |
|
||||
| Token model | `steward/models/api_tokens.py` | `ApiToken` |
|
||||
| Mint, resolve, revoke tokens | `steward/core/api_tokens.py` | `create_token()`, `resolve_token()`, `revoke_token()` |
|
||||
| Token management page and connect card | `steward/ai_access/routes.py`, `steward/templates/ai_access/` | `ai_access_bp` |
|
||||
|
||||
@@ -49,16 +49,6 @@ def get_blueprint():
|
||||
return docker_bp
|
||||
|
||||
|
||||
def register_mcp_tools(register) -> None:
|
||||
from .mcp_tools import register_tools
|
||||
register_tools(register)
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
from .mcp_tools import mcp_overview as findings
|
||||
return await findings(db)
|
||||
|
||||
|
||||
def get_nav() -> list[dict]:
|
||||
# Surfaces in the sidebar's Infrastructure group (fleet-wide container view).
|
||||
return [{"label": "Docker", "href": "/plugins/docker/"}]
|
||||
|
||||
@@ -92,55 +92,6 @@ def _derive_events(old_state: dict, new_containers: list) -> list:
|
||||
return events
|
||||
|
||||
|
||||
def _log_rows(log_batches, host_id: str):
|
||||
"""Pure: flatten the agent's log batches → docker_logs row kwargs (no DB).
|
||||
|
||||
`log_batches` is a list of (recorded_at, records) where each record is
|
||||
{"container", "stream", "ts", "line"}. A line's own Docker timestamp wins;
|
||||
recorded_at is the fallback when the agent couldn't parse one. Drops the
|
||||
agent's advisory truncation marker (container "_steward" — it signals a
|
||||
deferral, not a real container), malformed records, and lineless records;
|
||||
normalises an unknown stream to stdout. Unit-testable in isolation.
|
||||
"""
|
||||
for recorded_at, records in log_batches:
|
||||
if not isinstance(records, list):
|
||||
continue
|
||||
for rec in records:
|
||||
if not isinstance(rec, dict):
|
||||
continue
|
||||
name = rec.get("container")
|
||||
line = rec.get("line")
|
||||
if not name or name == "_steward" or line is None:
|
||||
continue
|
||||
ts = _parse_started_at(rec.get("ts")) or recorded_at
|
||||
stream = rec.get("stream")
|
||||
if stream not in ("stdout", "stderr"):
|
||||
stream = "stdout"
|
||||
yield {"host_id": host_id, "container_name": str(name)[:255],
|
||||
"ts": ts, "stream": stream, "line": str(line)}
|
||||
|
||||
|
||||
async def _persist_logs(session, host, log_batches) -> None:
|
||||
"""Append pushed container log lines (one row per line) for this host.
|
||||
|
||||
Time-series / append-only — the per-container size+age ring (retention)
|
||||
bounds growth, so a chatty container just keeps a shorter window. The global
|
||||
toggle + exclude list (Settings, no restart) are enforced here: this is the
|
||||
authoritative drop point, since the push model has no channel to tell an
|
||||
agent to stop collecting.
|
||||
"""
|
||||
from steward.core.settings import get_setting
|
||||
from .models import DockerLog
|
||||
|
||||
if not await get_setting(session, "docker.logs.enabled"):
|
||||
return
|
||||
exclude = set(await get_setting(session, "docker.logs.exclude") or ())
|
||||
for row in _log_rows(log_batches, host.id):
|
||||
if row["container_name"] in exclude:
|
||||
continue
|
||||
session.add(DockerLog(**row))
|
||||
|
||||
|
||||
async def _persist_swarm(session, host, swarm: dict) -> None:
|
||||
"""Upsert this manager's swarm topology; drop rows no longer reported.
|
||||
|
||||
@@ -169,7 +120,6 @@ async def _persist_swarm(session, host, swarm: dict) -> None:
|
||||
row.running = int(s.get("running") or 0)
|
||||
row.image = s.get("image")
|
||||
row.placement_json = json.dumps(s.get("placement") or [])
|
||||
row.tasks_json = json.dumps(s.get("tasks") or [])
|
||||
row.updated_at = now
|
||||
stale_services = delete(DockerSwarmService).where(
|
||||
DockerSwarmService.host_id == host.id)
|
||||
@@ -251,8 +201,7 @@ async def _persist_disk(session, host, disk: dict) -> None:
|
||||
await session.execute(stale)
|
||||
|
||||
|
||||
async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
|
||||
logs=None) -> None:
|
||||
async def persist_host_docker(session, host, snapshots, swarm=None, disk=None) -> None:
|
||||
"""Upsert containers + time-series + lifecycle events + swarm for one host.
|
||||
|
||||
`snapshots` is a list of (recorded_at: datetime, containers: list[dict]) —
|
||||
@@ -262,9 +211,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
|
||||
alert pipeline, and lifecycle-event derivation. `swarm` is the newest
|
||||
sample's swarm object (or None off managers) — persisted when present.
|
||||
`disk` is the newest sample's /system/df summary (or None on Docker-less
|
||||
hosts) — persisted when present. `logs` is a list of (recorded_at, records)
|
||||
log batches (m79) — appended one row per line when present, independent of
|
||||
whether this sample also carried container metrics.
|
||||
hosts) — persisted when present.
|
||||
"""
|
||||
from steward.core.alerts import record_metric
|
||||
from .models import DockerContainer, DockerEvent, DockerMetric
|
||||
@@ -273,8 +220,6 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
|
||||
await _persist_swarm(session, host, swarm)
|
||||
if disk is not None:
|
||||
await _persist_disk(session, host, disk)
|
||||
if logs:
|
||||
await _persist_logs(session, host, logs)
|
||||
|
||||
if not snapshots:
|
||||
return
|
||||
|
||||
@@ -1,703 +0,0 @@
|
||||
"""The docker plugin's MCP tools and its get_overview findings.
|
||||
|
||||
Registered through register_mcp_tools() in __init__.py, so they exist only
|
||||
while the plugin is enabled. Reads go through service.py, the same functions
|
||||
the plugin's pages use.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.ansible.runner import trigger_run
|
||||
from steward.ansible.sources import BUILTIN_SOURCE_NAME
|
||||
from steward.mcp._context import current_caller, mcp_app
|
||||
from steward.mcp.tools._audit import audit_mcp
|
||||
from steward.mcp.tools._shape import (
|
||||
age_s, bounded, clamp_limit, db_session, iso, lookback, now_utc, pick, rnd, summarize,
|
||||
)
|
||||
from steward.mcp.tools.overview import finding
|
||||
from steward.models.ansible_inventory import AnsibleTarget
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from . import service as docker_svc
|
||||
from .dedup import dedup_by_container_id
|
||||
from .models import DockerDiskUsage, DockerSwarmNode, DockerSwarmService
|
||||
from .swarm_view import build_swarm_services, replaced_tasks, task_time
|
||||
|
||||
PROBLEMS = ("restarting", "crash_looping", "unhealthy", "oom_killed", "exited_nonzero")
|
||||
CRASH_LOOP_DEATHS = docker_svc.CRASH_LOOP_DEATHS
|
||||
LOG_LINE_CHARS = 2000
|
||||
|
||||
|
||||
# ── pure helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def container_problems(c, failures_1h: int, failed_24h: bool,
|
||||
crash_loop_deaths: int = CRASH_LOOP_DEATHS) -> list[str]:
|
||||
"""What is wrong with one container, most severe first; [] when nothing is."""
|
||||
status = (c.status or "").lower()
|
||||
out = []
|
||||
if status == "restarting":
|
||||
out.append("restarting")
|
||||
if failures_1h >= crash_loop_deaths:
|
||||
out.append("crash_looping")
|
||||
if status == "running" and (c.health or "") == "unhealthy":
|
||||
out.append("unhealthy")
|
||||
if c.oom_killed:
|
||||
out.append("oom_killed")
|
||||
if status in ("exited", "dead") and (c.exit_code or 0) != 0 and failed_24h:
|
||||
out.append("exited_nonzero")
|
||||
return out
|
||||
|
||||
|
||||
def problem_since(problems: list[str], deaths: list, hour_ago: datetime,
|
||||
unhealthy_at: datetime | None = None) -> datetime | None:
|
||||
"""When a container's worst problem began; None when the retained events
|
||||
cannot say. `deaths` are its die/oom events of the last 24h, newest first.
|
||||
|
||||
A restart or crash loop dates from its first death in the last hour, an
|
||||
OOM kill or error exit from the death that ended it, and a failing
|
||||
healthcheck from when its health last turned unhealthy.
|
||||
"""
|
||||
if not problems:
|
||||
return None
|
||||
worst = problems[0]
|
||||
if worst in ("restarting", "crash_looping"):
|
||||
in_hour = [e.at for e in deaths if e.at >= hour_ago]
|
||||
return min(in_hour) if in_hour else (deaths[0].at if deaths else None)
|
||||
if worst == "unhealthy":
|
||||
return unhealthy_at
|
||||
if worst == "oom_killed":
|
||||
ooms = [e.at for e in deaths if e.event == "oom"]
|
||||
return ooms[0] if ooms else (deaths[0].at if deaths else None)
|
||||
return deaths[0].at if deaths else None
|
||||
|
||||
|
||||
def cut_middle(text: str, limit: int = LOG_LINE_CHARS) -> str:
|
||||
"""Keep both ends of an over-long line and say how much of the middle went."""
|
||||
if text is None or len(text) <= limit:
|
||||
return text
|
||||
keep = limit // 2
|
||||
return f"{text[:keep]} [… {len(text) - 2 * keep} characters omitted …] {text[-keep:]}"
|
||||
|
||||
|
||||
def container_row(c, host_names: dict, problems: list[str], now: datetime,
|
||||
replaced: bool = False, since: datetime | None = None) -> dict:
|
||||
running = (c.status or "").lower() == "running"
|
||||
return {
|
||||
"container_name": c.name,
|
||||
"host_id": c.host_id,
|
||||
"host_name": host_names.get(c.host_id),
|
||||
"image": c.image,
|
||||
"status": c.status,
|
||||
"health": c.health,
|
||||
"problems": problems,
|
||||
# When the worst problem began, from the container's retained events.
|
||||
"problem_since": iso(since),
|
||||
"problem_age_s": age_s(since, now),
|
||||
"exit_code": c.exit_code,
|
||||
"oom_killed": bool(c.oom_killed),
|
||||
"restart_count": c.restart_count,
|
||||
"cpu_pct": rnd(c.cpu_pct),
|
||||
"mem_pct": rnd(c.mem_pct),
|
||||
"mem_usage_bytes": c.mem_usage_bytes,
|
||||
"mem_limit_bytes": c.mem_limit_bytes,
|
||||
"started_at": iso(c.started_at),
|
||||
"uptime_s": age_s(c.started_at, now) if running and c.started_at else None,
|
||||
"compose_project": c.compose_project,
|
||||
"swarm_service": c.service_name,
|
||||
# A swarm task Swarm has replaced: history, judged on its service.
|
||||
"swarm_replaced": replaced if c.service_name else None,
|
||||
"reported_at": iso(c.scraped_at),
|
||||
"reported_age_s": age_s(c.scraped_at, now),
|
||||
}
|
||||
|
||||
|
||||
_SEVERITY = {"restarting": "critical", "crash_looping": "critical", "unhealthy": "critical",
|
||||
"oom_killed": "warning", "exited_nonzero": "warning"}
|
||||
_SAYS = {"restarting": "is restarting", "crash_looping": "keeps dying",
|
||||
"unhealthy": "fails its healthcheck", "oom_killed": "was OOM-killed",
|
||||
"exited_nonzero": "exited with an error"}
|
||||
|
||||
|
||||
def _from_iso(text: str | None) -> datetime | None:
|
||||
return datetime.fromisoformat(text.replace("Z", "+00:00")) if text else None
|
||||
|
||||
|
||||
def container_findings(rows: list[dict]) -> list[dict]:
|
||||
out = []
|
||||
for r in rows:
|
||||
if not r["problems"]:
|
||||
continue
|
||||
worst = r["problems"][0]
|
||||
where = f" on {r['host_name']}" if r["host_name"] else ""
|
||||
detail = f" (exit {r['exit_code']})" if r["exit_code"] not in (None, 0) else ""
|
||||
out.append(finding(
|
||||
_SEVERITY[worst], f"container_{worst}", r["container_name"],
|
||||
f"container {r['container_name']}{where} {' and '.join(_SAYS[p] for p in r['problems'])}{detail}",
|
||||
since=_from_iso(r["problem_since"]),
|
||||
ids={"host_id": r["host_id"], "container_name": r["container_name"]},
|
||||
next=f'get_container(container="{r["container_name"]}", host="{r["host_id"]}")'))
|
||||
return out
|
||||
|
||||
|
||||
def task_error_text(err: dict | None) -> str:
|
||||
"""Swarm's own account of a task's end, as one clause."""
|
||||
if not err:
|
||||
return ""
|
||||
why = err.get("error") or err.get("message") or err.get("state") or "unknown"
|
||||
code = f", exit {err['exit_code']}" if err.get("exit_code") not in (None, 0) else ""
|
||||
where = f" on {err['node']}" if err.get("node") else ""
|
||||
return f"{why}{code}{where}"
|
||||
|
||||
|
||||
def service_findings(services: list[dict]) -> list[dict]:
|
||||
"""Swarm services that are failing: crash-looping, under-replicated, or that
|
||||
lost a task to an error in the last 24h. Each names Swarm's last task error."""
|
||||
out = []
|
||||
for s in services:
|
||||
replicas = f"{s['running']} of {s['desired']} replicas running"
|
||||
last = task_error_text(s.get("last_error"))
|
||||
tail = f"; last task error: {last}" if last else ""
|
||||
ids = {"service_name": s["name"]}
|
||||
since = s.get("trouble_since")
|
||||
if s.get("crash_looping"):
|
||||
out.append(finding(
|
||||
"critical", "swarm_service_crash_looping", s["name"],
|
||||
f"swarm service {s['name']} keeps failing: {s['failures_1h']} tasks failed in the "
|
||||
f"last hour, {replicas}{tail}", since=since, ids=ids,
|
||||
next="list_swarm_services()"))
|
||||
elif s["state"] != "healthy":
|
||||
out.append(finding(
|
||||
"critical" if s["state"] == "down" else "warning", f"swarm_service_{s['state']}",
|
||||
s["name"], f"swarm service {s['name']} has {replicas}{tail}",
|
||||
since=since, ids=ids, next="list_swarm_services()"))
|
||||
elif s.get("failures_24h"):
|
||||
n = s["failures_24h"]
|
||||
out.append(finding(
|
||||
"warning", "swarm_task_failed", s["name"],
|
||||
f"swarm service {s['name']} had {n} failed task{'s' if n != 1 else ''} in the last "
|
||||
f"24h and is now {replicas}{tail}", since=since, ids=ids,
|
||||
next="list_swarm_services()"))
|
||||
return out
|
||||
|
||||
|
||||
# ── loading ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _host_names(db) -> dict[str, str]:
|
||||
return dict((await db.execute(select(Host.id, Host.name))).all())
|
||||
|
||||
|
||||
def _crash_loop_deaths() -> int:
|
||||
return docker_svc.crash_loop_deaths(mcp_app().config)
|
||||
|
||||
|
||||
async def _replaced(db, containers) -> set[tuple]:
|
||||
services = list((await db.execute(select(DockerSwarmService))).scalars())
|
||||
return replaced_tasks(containers, services)
|
||||
|
||||
|
||||
async def _rows(db, now: datetime) -> list[dict]:
|
||||
"""Every container with its problems. A swarm task Swarm replaced has none:
|
||||
its failure, if it was one, belongs to its service (service_findings)."""
|
||||
containers = dedup_by_container_id(await docker_svc.list_containers(db))
|
||||
replaced = await _replaced(db, containers)
|
||||
failures = await docker_svc.recent_failures_by_container(db, now - timedelta(hours=24))
|
||||
hour_ago = now - timedelta(hours=1)
|
||||
host_names = await _host_names(db)
|
||||
deaths = _crash_loop_deaths()
|
||||
sick = await docker_svc.unhealthy_since(
|
||||
db, [(c.host_id, c.name) for c in containers if (c.health or "") == "unhealthy"])
|
||||
rows = []
|
||||
for c in containers:
|
||||
key = (c.host_id, c.name)
|
||||
gone = key in replaced
|
||||
events = failures.get(key, [])
|
||||
recent = sum(1 for e in events if e.at >= hour_ago)
|
||||
problems = [] if gone else container_problems(c, recent, bool(events), deaths)
|
||||
since = problem_since(problems, events, hour_ago, sick.get(key))
|
||||
rows.append(container_row(c, host_names, problems, now, replaced=gone, since=since))
|
||||
return rows
|
||||
|
||||
|
||||
async def _swarm(db, now: datetime):
|
||||
containers = dedup_by_container_id(await docker_svc.list_containers(db))
|
||||
services = list((await db.execute(select(DockerSwarmService))).scalars())
|
||||
nodes = list((await db.execute(select(DockerSwarmNode))).scalars())
|
||||
host_names = await _host_names(db)
|
||||
node_hostname = {n.node_id: n.hostname for n in nodes if n.hostname}
|
||||
built = build_swarm_services(containers, services, node_hostname, host_names,
|
||||
now=now, crash_loop_deaths=_crash_loop_deaths())
|
||||
return built["services"], nodes, host_names
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
"""get_overview findings: failing containers and under-replicated swarm services."""
|
||||
now = now_utc()
|
||||
rows = await _rows(db, now)
|
||||
services, _, _ = await _swarm(db, now)
|
||||
return container_findings(rows) + service_findings(services)
|
||||
|
||||
|
||||
# ── tools ────────────────────────────────────────────────────────────────────
|
||||
|
||||
async def list_containers(host: str = "", status: str = "", problem: str = "",
|
||||
compose_project: str = "", limit: int = 100) -> dict:
|
||||
"""Docker containers across every host that runs a Steward agent with Docker.
|
||||
|
||||
host: a host's name or id (list_hosts); empty for all hosts.
|
||||
status: Docker's state — "running", "exited", "restarting", "paused",
|
||||
"created" or "dead"; empty for all.
|
||||
problem: "any" for containers with something wrong, or one of
|
||||
"restarting", "crash_looping" (died repeatedly within the last hour;
|
||||
the count is set under Settings → Thresholds, default 3), "unhealthy"
|
||||
(failing its healthcheck), "oom_killed", "exited_nonzero" (died with an
|
||||
error in the last 24h); empty for no filter.
|
||||
compose_project: only containers from this Compose project.
|
||||
limit: default 100, max 500.
|
||||
|
||||
Containers with problems come first. Each row's `problems` lists what is
|
||||
wrong, most severe first, and `problem_since` / `problem_age_s` say when
|
||||
the worst of them began (null when Steward's retained events cannot say).
|
||||
`reported_age_s` is how old the agent's last report is, so a large value
|
||||
means the row may be stale.
|
||||
|
||||
A swarm task's container that Swarm has replaced (a rolling update, or a
|
||||
retry after a failure) has `swarm_replaced: true` and no problems: Swarm
|
||||
restarts a task as a new container, so failures are counted on the
|
||||
service — list_swarm_services shows them.
|
||||
"""
|
||||
if problem and problem != "any" and problem not in PROBLEMS:
|
||||
raise ValueError(f"problem {problem!r} is not 'any' or one of {', '.join(PROBLEMS)}.")
|
||||
limit = clamp_limit(limit, 100, 500)
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
rows = await _rows(db, now)
|
||||
if host:
|
||||
hosts = list((await db.execute(select(Host))).scalars())
|
||||
h = pick(host, hosts, kind="host", list_tool="list_hosts")
|
||||
rows = [r for r in rows if r["host_id"] == h.id]
|
||||
if status:
|
||||
rows = [r for r in rows if (r["status"] or "").lower() == status.lower()]
|
||||
if problem == "any":
|
||||
rows = [r for r in rows if r["problems"]]
|
||||
elif problem:
|
||||
rows = [r for r in rows if problem in r["problems"]]
|
||||
if compose_project:
|
||||
rows = [r for r in rows if r["compose_project"] == compose_project]
|
||||
rows.sort(key=lambda r: (not r["problems"], (r["status"] or "") != "running",
|
||||
(r["host_name"] or "").lower(), r["container_name"]))
|
||||
return bounded(rows, limit, key="containers",
|
||||
more="Pass a larger `limit` (max 500), or filter by `host`, `status` or `problem`.")
|
||||
|
||||
|
||||
@dataclass
|
||||
class RemovedContainer:
|
||||
"""A container Docker has removed, known only by its retained logs and events."""
|
||||
host_id: str
|
||||
name: str
|
||||
last_seen: datetime
|
||||
|
||||
|
||||
_REMOVED_HINT = ("It may have been removed: Swarm replaces a dead task with a new container under "
|
||||
"a new name. list_containers(status=\"exited\") lists stopped containers, and "
|
||||
"search_container_logs(host=...) searches a whole host.")
|
||||
|
||||
|
||||
async def _pick_container(db, container: str, host: str, *, removed: bool = False):
|
||||
"""The container named `container`: a live one, else (with `removed`) one
|
||||
Docker has removed whose logs or events Steward still holds, returned as a
|
||||
RemovedContainer. Exact names beat partial ones, and live beats removed."""
|
||||
containers = await docker_svc.list_containers(db)
|
||||
host_id = None
|
||||
if host:
|
||||
hosts = list((await db.execute(select(Host))).scalars())
|
||||
host_id = pick(host, hosts, kind="host", list_tool="list_hosts").id
|
||||
containers = [c for c in containers if c.host_id == host_id]
|
||||
want = (container or "").strip()
|
||||
if not want:
|
||||
raise ValueError("pass a container name; list_containers lists them.")
|
||||
live = {(c.host_id, c.name) for c in containers}
|
||||
|
||||
def _one(found, what):
|
||||
if len(found) > 1:
|
||||
raise ValueError(f"{len(found)} hosts have {what} named {want!r}; pass `host` "
|
||||
f"(one of: {', '.join(sorted(f[0] for f in found))}).")
|
||||
return found[0] if found else None
|
||||
|
||||
same = [c for c in containers if c.name == want] or \
|
||||
[c for c in containers if c.name.lower() == want.lower()]
|
||||
if same:
|
||||
return _one([(c.host_id, c) for c in same], "a container")[1]
|
||||
if removed:
|
||||
gone = [r for r in await docker_svc.retained_names(db, want, exact=True, host_id=host_id)
|
||||
if (r[0], r[1]) not in live]
|
||||
if gone:
|
||||
return RemovedContainer(*_one(gone, "a removed container"))
|
||||
# One more chance: a unique container whose name contains it (swarm task
|
||||
# names look like "stack_svc.1.k2j…").
|
||||
partial = [c for c in containers if want.lower() in c.name.lower()]
|
||||
if len(partial) == 1:
|
||||
return partial[0]
|
||||
gone = []
|
||||
if removed and not partial:
|
||||
gone = [r for r in await docker_svc.retained_names(db, want, exact=False, host_id=host_id)
|
||||
if (r[0], r[1]) not in live]
|
||||
if len(gone) == 1:
|
||||
return RemovedContainer(*gone[0])
|
||||
names = sorted(c.name for c in partial)[:10] + [f"{r[1]} (removed)" for r in gone[:10]]
|
||||
hint = f" Containers whose names contain it: {', '.join(names)}." if names else ""
|
||||
raise ValueError(f"no container named {want!r}{' on that host' if host else ''}.{hint} "
|
||||
f"{'' if names else _REMOVED_HINT + ' '}list_containers lists them.")
|
||||
|
||||
|
||||
def _removed_note(c: RemovedContainer) -> str:
|
||||
return (f"{c.name} no longer exists: Docker removed it (Swarm replaces a failed task with a new "
|
||||
f"container). This is what Steward retained; it was last seen {iso(c.last_seen)}.")
|
||||
|
||||
|
||||
async def get_container(container: str, host: str = "", range: str = "24h", events: int = 20) -> dict:
|
||||
"""One container in depth: state, resources over time and lifecycle events.
|
||||
|
||||
container: the container's name, as list_containers shows. A unique part of
|
||||
a name also works (swarm task names are long).
|
||||
host: the host's name or id. Needed only when several hosts run a
|
||||
container with this name.
|
||||
range: the window for `cpu_pct_history` and `mem_pct_history` (min, avg,
|
||||
max, p95, last and a ≤60-point series) — "1h", "6h", "24h" (default),
|
||||
"7d" or "30d". `cpu_pct` and `mem_pct` are the latest reported values.
|
||||
events: how many lifecycle events (start, stop, die, oom, health_change)
|
||||
to return, newest first (default 20, max 100).
|
||||
|
||||
A container Docker has since removed (a dead Swarm task, a `docker rm`)
|
||||
still resolves by its exact name, or by a part of it that matches one
|
||||
removed container, while Steward retains its logs or events: the reply
|
||||
has `removed: true`, `last_seen_at`, its history and its events, and no
|
||||
live state. A live container has `removed: false`.
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
events = clamp_limit(events, 20, 100)
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
c = await _pick_container(db, container, host, removed=True)
|
||||
if isinstance(c, RemovedContainer):
|
||||
return await _removed_container(db, c, since, key, events, now)
|
||||
gone = (c.host_id, c.name) in await _replaced(db, await docker_svc.list_containers(db))
|
||||
failures = (await docker_svc.recent_failures_by_container(
|
||||
db, now - timedelta(hours=24))).get((c.host_id, c.name), [])
|
||||
host_names = await _host_names(db)
|
||||
cpu = await docker_svc.container_points(db, c.host_id, c.name, "cpu_pct", since)
|
||||
mem = await docker_svc.container_points(db, c.host_id, c.name, "mem_pct", since)
|
||||
evs = await docker_svc.container_events(db, c.host_id, c.name, events)
|
||||
sick = await docker_svc.unhealthy_since(db, [(c.host_id, c.name)])
|
||||
hour_ago = now - timedelta(hours=1)
|
||||
recent = sum(1 for e in failures if e.at >= hour_ago)
|
||||
try:
|
||||
ports = json.loads(c.ports_json or "[]")
|
||||
except ValueError:
|
||||
ports = []
|
||||
problems = [] if gone else container_problems(c, recent, bool(failures), _crash_loop_deaths())
|
||||
began = problem_since(problems, failures, hour_ago, sick.get((c.host_id, c.name)))
|
||||
return {
|
||||
**container_row(c, host_names, problems, now, replaced=gone, since=began),
|
||||
"removed": False,
|
||||
"container_id": c.container_id,
|
||||
"ports": ports,
|
||||
"net_rx_bytes": c.net_rx_bytes, "net_tx_bytes": c.net_tx_bytes,
|
||||
"blk_read_bytes": c.blk_read_bytes, "blk_write_bytes": c.blk_write_bytes,
|
||||
"range": key,
|
||||
"cpu_pct_history": summarize(cpu),
|
||||
"mem_pct_history": summarize(mem),
|
||||
"events": _event_rows(evs, now),
|
||||
"next": f'search_container_logs(container="{c.name}", host="{c.host_id}")',
|
||||
}
|
||||
|
||||
|
||||
def _event_rows(evs, now: datetime) -> list[dict]:
|
||||
return [{"event": e.event, "detail": e.detail, "at": iso(e.at), "age_s": age_s(e.at, now)}
|
||||
for e in evs]
|
||||
|
||||
|
||||
async def _removed_container(db, c: RemovedContainer, since, key: str, events: int,
|
||||
now: datetime) -> dict:
|
||||
"""get_container for a container Docker has removed: what Steward kept."""
|
||||
host_names = await _host_names(db)
|
||||
cpu = await docker_svc.container_points(db, c.host_id, c.name, "cpu_pct", since)
|
||||
mem = await docker_svc.container_points(db, c.host_id, c.name, "mem_pct", since)
|
||||
evs = await docker_svc.container_events(db, c.host_id, c.name, events)
|
||||
return {
|
||||
"container_name": c.name, "host_id": c.host_id, "host_name": host_names.get(c.host_id),
|
||||
"removed": True, "note": _removed_note(c),
|
||||
"last_seen_at": iso(c.last_seen), "last_seen_age_s": age_s(c.last_seen, now),
|
||||
"range": key,
|
||||
"cpu_pct_history": summarize(cpu),
|
||||
"mem_pct_history": summarize(mem),
|
||||
"events": _event_rows(evs, now),
|
||||
"next": f'search_container_logs(container="{c.name}", host="{c.host_id}", range="24h")',
|
||||
}
|
||||
|
||||
|
||||
async def list_swarm_services(limit: int = 100) -> dict:
|
||||
"""Docker Swarm services with their replicas per node, and the swarm's nodes.
|
||||
|
||||
`state` is "healthy" (all desired replicas running), "degraded" (some) or
|
||||
"down" (none). `replicas` are the service's current tasks: each names its
|
||||
host and container status, and one marked `ghost` runs on a node with no
|
||||
Steward agent, so Steward knows only how many tasks are placed there.
|
||||
Tasks Swarm replaced (rolling updates, retries) are left out and counted
|
||||
in `previous_tasks`.
|
||||
|
||||
From Swarm's own task records (a manager's agent 1.8.0 or later;
|
||||
`tasks_known: false` and nulls otherwise):
|
||||
- `failures_1h` / `failures_24h`: tasks that failed or were rejected.
|
||||
Swarm retries a failed task as a new one, so this, not any container's
|
||||
restart count, is how a crash loop shows. Swarm keeps about 5 old tasks
|
||||
per replica, which bounds the count.
|
||||
- `crash_looping`: failures_1h reached the crash-loop count set under
|
||||
Settings → Thresholds (default 3).
|
||||
- `last_error`: the newest task that failed, was rejected, or carries an
|
||||
error while Swarm retries it — `error` is Swarm's own text (e.g. "No
|
||||
such image", "task: non-zero exit (1)"), with `exit_code`, `node`, `at`.
|
||||
- `trouble_since` / `trouble_age_s`: when the current trouble began — a
|
||||
crash loop's first failure in the hour, a short service's first failed
|
||||
task in its failing slots, a recovered service's first failure in 24h.
|
||||
Swarm keeps few old tasks, so a long outage reads as a lower bound.
|
||||
|
||||
Services with a problem come first. `nodes` lists every swarm node once,
|
||||
with its role (manager/worker), availability and status.
|
||||
limit: default 100, max 500.
|
||||
"""
|
||||
limit = clamp_limit(limit, 100, 500)
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
services, nodes, host_names = await _swarm(db, now)
|
||||
seen, node_rows = set(), []
|
||||
for n in sorted(nodes, key=lambda n: n.updated_at, reverse=True):
|
||||
if n.node_id in seen:
|
||||
continue
|
||||
seen.add(n.node_id)
|
||||
node_rows.append({"node_id": n.node_id, "hostname": n.hostname, "role": n.role,
|
||||
"leader": n.leader, "availability": n.availability, "status": n.status,
|
||||
"reported_by": host_names.get(n.host_id)})
|
||||
node_rows.sort(key=lambda n: n["hostname"].lower())
|
||||
order = {"down": 0, "degraded": 1, "healthy": 2}
|
||||
|
||||
def _rank(s):
|
||||
return (not s["crash_looping"], order.get(s["state"], 3), not s["failures_24h"], s["name"])
|
||||
|
||||
def _error_row(err):
|
||||
if err is None:
|
||||
return None
|
||||
at = task_time(err)
|
||||
return {"error": err["error"], "message": err["message"], "state": err["state"],
|
||||
"exit_code": err["exit_code"], "node": err["node"],
|
||||
"at": iso(at), "age_s": age_s(at, now) if at else None}
|
||||
|
||||
rows = [
|
||||
{
|
||||
"service_name": s["name"], "mode": s["mode"], "state": s["state"],
|
||||
"running": s["running"], "desired": s["desired"],
|
||||
"crash_looping": s["crash_looping"],
|
||||
"failures_1h": s["failures_1h"], "failures_24h": s["failures_24h"],
|
||||
"last_error": _error_row(s["last_error"]),
|
||||
"trouble_since": iso(s["trouble_since"]),
|
||||
"trouble_age_s": age_s(s["trouble_since"], now),
|
||||
"previous_tasks": s["previous_tasks"], "tasks_known": s["tasks_known"],
|
||||
"replicas": [
|
||||
{"ghost": True, "host": r["host"], "count": r["count"]} if r["ghost"] else
|
||||
{"ghost": False, "host": r["host"], "host_id": r["host_id"], "container_name": r["name"],
|
||||
"status": r["status"], "health": r["health"], "restart_count": r["restart_count"],
|
||||
"cpu_pct": rnd(r["cpu_pct"]), "mem_pct": rnd(r["mem_pct"])}
|
||||
for r in s["replicas"]
|
||||
],
|
||||
}
|
||||
for s in sorted(services, key=_rank)
|
||||
]
|
||||
return {"nodes": node_rows, **bounded(rows, limit, key="services")}
|
||||
|
||||
|
||||
async def search_container_logs(container: str = "", host: str = "", text: str = "",
|
||||
range: str = "1h", stream: str = "", before: str = "",
|
||||
context: int = 0, limit: int = 50) -> dict:
|
||||
"""Search the container log lines Steward has retained, newest first.
|
||||
|
||||
container: one container's name (as list_containers shows); empty searches
|
||||
every container (narrow with `host`). A container Docker has removed
|
||||
still works while its lines are retained: the reply then has
|
||||
`container_removed: true` and a `note` saying when it was last seen.
|
||||
host: a host's name or id; empty for all hosts.
|
||||
text: case-insensitive substring to match; empty matches every line.
|
||||
range: how far back — "1h" (default), "6h", "24h", "7d" or "30d". Logs are
|
||||
kept per container up to a size and age limit, so older lines may be gone.
|
||||
stream: "stdout" or "stderr"; empty for both.
|
||||
before: an ISO time; returns only lines older than it. Pass a response's
|
||||
`next_before` to page further back.
|
||||
context: lines of surrounding log (0-5) to include with each match, from
|
||||
the same container. Only with `container` set.
|
||||
limit: matches to return, default 50, max 200.
|
||||
|
||||
Lines longer than 2000 characters keep both ends, and the middle is
|
||||
replaced by a marker saying how much was cut.
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
limit = clamp_limit(limit, 50, 200)
|
||||
context = max(0, min(int(context or 0), 5))
|
||||
if context and not container:
|
||||
raise ValueError("context needs `container`: it shows lines around a match within one container's log.")
|
||||
if stream and stream not in ("stdout", "stderr"):
|
||||
raise ValueError(f"stream {stream!r} is not 'stdout' or 'stderr'.")
|
||||
before_dt = None
|
||||
if before:
|
||||
try:
|
||||
before_dt = datetime.fromisoformat(before.replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"before {before!r} is not an ISO-8601 time; pass a response's next_before.") from exc
|
||||
async with db_session() as db:
|
||||
host_id = name = removed = None
|
||||
if container:
|
||||
c = await _pick_container(db, container, host, removed=True)
|
||||
host_id, name = c.host_id, c.name
|
||||
removed = c if isinstance(c, RemovedContainer) else None
|
||||
elif host:
|
||||
hosts = list((await db.execute(select(Host))).scalars())
|
||||
host_id = pick(host, hosts, kind="host", list_tool="list_hosts").id
|
||||
rows = await docker_svc.search_logs(db, since=since, host_id=host_id, name=name, text=text,
|
||||
stream=stream, before=before_dt, limit=limit + 1)
|
||||
more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
host_names = await _host_names(db)
|
||||
matches = []
|
||||
for r in rows:
|
||||
m = {"at": iso(r.ts), "container_name": r.container_name, "host_id": r.host_id,
|
||||
"host_name": host_names.get(r.host_id), "stream": r.stream, "line": cut_middle(r.line)}
|
||||
if context:
|
||||
pre, post = await docker_svc.log_context(db, r.host_id, r.container_name, r.ts, context)
|
||||
m["before"] = [cut_middle(x.line) for x in pre]
|
||||
m["after"] = [cut_middle(x.line) for x in post]
|
||||
matches.append(m)
|
||||
out = {"range": key, "matches": matches, "returned": len(matches), "truncated": more}
|
||||
if removed is not None:
|
||||
out["container_removed"] = True
|
||||
out["note"] = _removed_note(removed)
|
||||
if removed.last_seen < since:
|
||||
out["note"] += " That is before this range; pass a longer `range`."
|
||||
if more:
|
||||
# Full precision: `at` is cut to the second for reading, and a cursor
|
||||
# cut the same way would skip lines logged earlier in that second.
|
||||
out["next_before"] = rows[-1].ts.isoformat()
|
||||
out["more"] = "More matches are older; pass next_before as `before`, or narrow with `text`."
|
||||
return out
|
||||
|
||||
|
||||
async def get_docker_disk_usage(host: str = "") -> dict:
|
||||
"""Docker disk usage per host: images, containers, volumes and build cache, in bytes.
|
||||
|
||||
host: a host's name or id; empty for every host. `images_reclaimable_bytes`
|
||||
is what pruning unused images would free. An admin can prune from
|
||||
Steward's Docker → Disk page.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
summaries = list((await db.execute(select(DockerDiskUsage))).scalars())
|
||||
host_names = await _host_names(db)
|
||||
if host:
|
||||
hosts = list((await db.execute(select(Host))).scalars())
|
||||
hid = pick(host, hosts, kind="host", list_tool="list_hosts").id
|
||||
summaries = [s for s in summaries if s.host_id == hid]
|
||||
rows = [
|
||||
{
|
||||
"host_id": s.host_id, "host_name": host_names.get(s.host_id),
|
||||
"images_total": s.images_total, "images_active": s.images_active,
|
||||
"images_bytes": s.images_size, "images_reclaimable_bytes": s.images_reclaimable,
|
||||
"containers_count": s.containers_count, "containers_bytes": s.containers_size,
|
||||
"volumes_count": s.volumes_count, "volumes_bytes": s.volumes_size,
|
||||
"build_cache_bytes": s.build_cache_size,
|
||||
"reported_at": iso(s.updated_at), "reported_age_s": age_s(s.updated_at),
|
||||
}
|
||||
for s in summaries
|
||||
]
|
||||
rows.sort(key=lambda r: -(r["images_reclaimable_bytes"] or 0))
|
||||
return {"hosts": rows, "total": len(rows)}
|
||||
|
||||
|
||||
PRUNE_BUILTIN_PLAYBOOK = "maintenance/docker_prune.yml"
|
||||
_STOPPED = ("exited", "created", "dead")
|
||||
|
||||
|
||||
def prune_preview(what: str, usage, stopped: int) -> dict:
|
||||
"""What a prune of `what` would remove, from the host's latest disk report."""
|
||||
out: dict = {}
|
||||
if what in ("containers", "system"):
|
||||
out["stopped_containers"] = stopped
|
||||
if what == "images":
|
||||
out["unused_images_bytes"] = usage.images_reclaimable if usage else None
|
||||
if what == "system":
|
||||
out["dangling_images"] = "yes (unused but tagged images stay)"
|
||||
out["build_cache_bytes"] = usage.build_cache_size if usage else None
|
||||
out["unused_networks"] = "yes"
|
||||
return out
|
||||
|
||||
|
||||
async def prune_docker(host: str, what: str, check: bool = True) -> dict:
|
||||
"""Reclaim Docker disk on a host. Previews unless check=false.
|
||||
|
||||
host: the Steward host's name or id. It needs a linked Ansible target
|
||||
(list_inventory shows the links); the prune runs through Ansible as
|
||||
Steward's built-in docker_prune playbook.
|
||||
what: "containers" removes stopped containers; "images" removes ALL
|
||||
unused images, not only dangling ones; "system" is `docker system
|
||||
prune -f`: stopped containers, dangling images, unused networks and
|
||||
build cache. Volumes are never pruned.
|
||||
check: true (default) starts nothing and returns `would_remove`, sized
|
||||
from the host's latest Docker disk report. Ansible's own check mode
|
||||
skips the prune commands, so a check-mode run would show nothing.
|
||||
Pass false to run the prune.
|
||||
|
||||
Admin only, as on the Docker → Disk page. A real prune returns `run_id`
|
||||
(follow it with wait_for_run) and is recorded in the audit log.
|
||||
"""
|
||||
what = (what or "").strip()
|
||||
if what not in docker_svc.PRUNE_TARGETS:
|
||||
raise ValueError(f"what {what!r} is not one of {', '.join(docker_svc.PRUNE_TARGETS)}.")
|
||||
async with db_session() as db:
|
||||
h = pick(host, (await db.execute(select(Host))).scalars(), kind="host", list_tool="list_hosts")
|
||||
target = (await db.execute(
|
||||
select(AnsibleTarget).where(AnsibleTarget.host_id == h.id))).scalars().first()
|
||||
usage = await db.get(DockerDiskUsage, h.id)
|
||||
stopped = sum(1 for c in await docker_svc.list_containers(db, h.id)
|
||||
if (c.status or "").lower() in _STOPPED)
|
||||
if target is None:
|
||||
raise ValueError(f"host {h.name!r} has no linked Ansible target, so Steward cannot reach it "
|
||||
"to prune. Link one under Ansible → Inventory. Nothing was run.")
|
||||
preview = {
|
||||
"host": h.name,
|
||||
"what": what,
|
||||
"ansible_target": target.name,
|
||||
"would_remove": prune_preview(what, usage, stopped),
|
||||
"disk_report_age_s": age_s(usage.updated_at) if usage else None,
|
||||
}
|
||||
if check:
|
||||
return {**preview, "check": True,
|
||||
"summary": f"Nothing was run. Pass check=false to prune {what} on {h.name}."}
|
||||
caller = current_caller()
|
||||
run, _src, err = await trigger_run(
|
||||
mcp_app(), source_name=BUILTIN_SOURCE_NAME, playbook_path=PRUNE_BUILTIN_PLAYBOOK,
|
||||
inventory_scope=f"steward:target:{target.id}",
|
||||
params={"extra_vars_map": docker_svc.prune_extra_vars(what)},
|
||||
triggered_by=caller.user_id)
|
||||
if err:
|
||||
raise ValueError(f"the prune could not start: {err}.")
|
||||
await audit_mcp("docker.prune.started", "host", h.name,
|
||||
{"host_id": h.id, "what": what, "run_id": run.id})
|
||||
return {**preview, "check": False, "run_id": run.id,
|
||||
"next": f'wait_for_run(run_id="{run.id}")'}
|
||||
|
||||
|
||||
def register_tools(register) -> None:
|
||||
for fn in (list_containers, get_container, list_swarm_services,
|
||||
search_container_logs, get_docker_disk_usage):
|
||||
register(fn, role=UserRole.viewer, read_only=True)
|
||||
register(prune_docker, role=UserRole.admin, read_only=False)
|
||||
@@ -1,48 +0,0 @@
|
||||
"""Docker container logs table
|
||||
|
||||
Adds docker_logs — one row per container log line, pushed by the host agent and
|
||||
folded into its metrics push. Time-series, host-scoped (container names are only
|
||||
unique within a host). Bounded by a per-container size+age ring in retention, so
|
||||
a chatty container keeps a shorter window rather than growing without limit.
|
||||
Additive create_table + twin indexes (viewer lookup by (host, container, ts);
|
||||
age-cutoff prune by ts).
|
||||
|
||||
Revision ID: docker_009_container_logs
|
||||
Revises: docker_008_bigint_mem
|
||||
Create Date: 2026-07-19
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "docker_009_container_logs"
|
||||
down_revision: Union[str, None] = "docker_008_bigint_mem"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"docker_logs",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("host_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("container_name", sa.String(length=255), nullable=False),
|
||||
sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("stream", sa.String(length=8), nullable=False, server_default="stdout"),
|
||||
sa.Column("line", sa.Text(), nullable=False, server_default=""),
|
||||
sa.ForeignKeyConstraint(["host_id"], ["hosts.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index("ix_docker_logs_host_id", "docker_logs", ["host_id"])
|
||||
op.create_index("ix_docker_logs_container_name", "docker_logs", ["container_name"])
|
||||
op.create_index("ix_docker_logs_host_container_time",
|
||||
"docker_logs", ["host_id", "container_name", "ts"])
|
||||
op.create_index("ix_docker_logs_ts", "docker_logs", ["ts"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_docker_logs_ts", table_name="docker_logs")
|
||||
op.drop_index("ix_docker_logs_host_container_time", table_name="docker_logs")
|
||||
op.drop_index("ix_docker_logs_container_name", table_name="docker_logs")
|
||||
op.drop_index("ix_docker_logs_host_id", table_name="docker_logs")
|
||||
op.drop_table("docker_logs")
|
||||
@@ -1,30 +0,0 @@
|
||||
"""Docker swarm service tasks column
|
||||
|
||||
Adds docker_swarm_services.tasks_json — each service's newest tasks as a
|
||||
manager's agent (≥ 1.8.0) reports them: slot, node, desired and current state,
|
||||
Swarm's error and message, exit code and time. Lets Steward tell a task Swarm
|
||||
replaced from one that crashed, count a service's failures across task
|
||||
retries, and say why its last task failed. Additive column; no DROP+recreate.
|
||||
|
||||
Revision ID: docker_010_swarm_tasks
|
||||
Revises: docker_009_container_logs
|
||||
Create Date: 2026-10-06
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "docker_010_swarm_tasks"
|
||||
down_revision: Union[str, None] = "docker_009_container_logs"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("docker_swarm_services",
|
||||
sa.Column("tasks_json", sa.Text, nullable=False,
|
||||
server_default="[]"))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("docker_swarm_services", "tasks_json")
|
||||
@@ -163,43 +163,6 @@ class DockerEvent(Base):
|
||||
)
|
||||
|
||||
|
||||
class DockerLog(Base):
|
||||
"""Container log lines pushed by the host agent — one row per line.
|
||||
|
||||
Time-series, scoped to the reporting host (container names are only unique
|
||||
within a host, same identity as docker_metrics). The agent tails each running
|
||||
container incrementally and folds new lines into its metrics push; `ts` is the
|
||||
line's own Docker timestamp. Bounded by a per-container size+age ring
|
||||
(retention), so a chatty container just keeps a shorter window rather than
|
||||
growing without limit.
|
||||
"""
|
||||
__tablename__ = "docker_logs"
|
||||
|
||||
id: Mapped[str] = mapped_column(
|
||||
String(36), primary_key=True, default=lambda: str(uuid.uuid4())
|
||||
)
|
||||
host_id: Mapped[str] = mapped_column(
|
||||
String(36), ForeignKey("hosts.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
container_name: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
|
||||
ts: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False,
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
)
|
||||
stream: Mapped[str] = mapped_column(String(8), nullable=False, default="stdout")
|
||||
# stdout | stderr
|
||||
line: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
# Viewer filters on (host_id, container_name) and sorts by time; the ring
|
||||
# prune walks the same key newest-first. A second index on `ts` alone serves
|
||||
# the age-cutoff delete. Twin-index idiom, mirroring docker_events.
|
||||
__table_args__ = (
|
||||
Index("ix_docker_logs_host_container_time",
|
||||
"host_id", "container_name", "ts"),
|
||||
Index("ix_docker_logs_ts", "ts"),
|
||||
)
|
||||
|
||||
|
||||
class DockerSwarmService(Base):
|
||||
"""A Swarm service as seen by a manager host: desired vs running replicas.
|
||||
|
||||
@@ -223,11 +186,6 @@ class DockerSwarmService(Base):
|
||||
# A manager sees every task, so this captures cross-node placement that the
|
||||
# local docker_containers rows (this host only) can't.
|
||||
placement_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
|
||||
# The service's newest tasks, newest first, as the manager's agent shaped
|
||||
# them: [{"task_id", "slot", "node_id", "desired_state", "state", "message",
|
||||
# "error", "exit_code", "at"}], JSON. Swarm's own record of which tasks it
|
||||
# replaced and why each one stopped; "[]" from agents older than 1.8.0.
|
||||
tasks_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False,
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
|
||||
@@ -38,8 +38,6 @@ async def run_docker_retention(
|
||||
events_days: int,
|
||||
metrics_raw_days: int,
|
||||
metrics_rollup_days: int,
|
||||
logs_retention_days: int = 3,
|
||||
logs_max_bytes_per_container: int = 5_000_000,
|
||||
now: datetime | None = None,
|
||||
) -> dict:
|
||||
"""Roll up + prune Docker time-series. Returns a counts dict for logging.
|
||||
@@ -49,22 +47,18 @@ async def run_docker_retention(
|
||||
re-run is idempotent, then delete those raw rows.
|
||||
2. Prune rolled-up rows older than the rollup window.
|
||||
3. Prune docker_events older than the events window.
|
||||
4. Prune docker_logs with a per-container size+age ring (m79): drop lines
|
||||
older than the age window, then keep only the newest ~cap bytes per
|
||||
(host, container).
|
||||
"""
|
||||
from datetime import timezone
|
||||
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
|
||||
from .models import DockerEvent, DockerLog, DockerMetric, DockerMetricHourly
|
||||
from .models import DockerEvent, DockerMetric, DockerMetricHourly
|
||||
|
||||
if now is None:
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
rolled = rolled_rows = events_pruned = rollup_pruned = 0
|
||||
logs_age_pruned = logs_size_pruned = 0
|
||||
|
||||
# ── 1. Roll up raw metrics older than the raw window into hourly buckets ──
|
||||
raw_cutoff = _rollup_cutoff(now, metrics_raw_days)
|
||||
@@ -126,35 +120,9 @@ async def run_docker_retention(
|
||||
)
|
||||
events_pruned = res.rowcount or 0
|
||||
|
||||
# ── 4. Container-log ring: age cutoff, then per-container byte cap (m79) ──
|
||||
logs_cutoff = now - timedelta(days=logs_retention_days)
|
||||
res = await session.execute(
|
||||
delete(DockerLog).where(DockerLog.ts < logs_cutoff)
|
||||
)
|
||||
logs_age_pruned = res.rowcount or 0
|
||||
|
||||
# Size ring: per (host, container), sum line bytes newest-first; delete a row
|
||||
# once its strictly-newer siblings already fill the cap. Using the EXCLUSIVE
|
||||
# prefix (running total minus this row) means the newest row always survives,
|
||||
# so a single line larger than the cap is never wiped out.
|
||||
running = func.sum(func.length(DockerLog.line)).over(
|
||||
partition_by=[DockerLog.host_id, DockerLog.container_name],
|
||||
order_by=[DockerLog.ts.desc(), DockerLog.id.desc()],
|
||||
)
|
||||
prefix_excl = (running - func.length(DockerLog.line)).label("prefix_excl")
|
||||
ranked = select(DockerLog.id, prefix_excl).subquery()
|
||||
over_cap = select(ranked.c.id).where(
|
||||
ranked.c.prefix_excl >= logs_max_bytes_per_container)
|
||||
res = await session.execute(
|
||||
delete(DockerLog).where(DockerLog.id.in_(over_cap))
|
||||
)
|
||||
logs_size_pruned = res.rowcount or 0
|
||||
|
||||
return {
|
||||
"buckets_rolled": rolled,
|
||||
"raw_rows_rolled": rolled_rows,
|
||||
"rollup_pruned": rollup_pruned,
|
||||
"events_pruned": events_pruned,
|
||||
"logs_age_pruned": logs_age_pruned,
|
||||
"logs_size_pruned": logs_size_pruned,
|
||||
}
|
||||
|
||||
+84
-140
@@ -3,34 +3,23 @@ from __future__ import annotations
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from quart import (
|
||||
Blueprint, current_app, jsonify, redirect, render_template, request,
|
||||
session, url_for,
|
||||
)
|
||||
from quart import Blueprint, current_app, render_template, request
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.models.users import UserRole
|
||||
from steward.models.hosts import Host
|
||||
from steward.core.time_range import parse_range, DEFAULT_RANGE
|
||||
from . import service as docker_svc
|
||||
from .dedup import dedup_by_container_id
|
||||
from .swarm_view import build_swarm_services, parse_tasks, task_summary
|
||||
from .swarm_view import build_swarm_services
|
||||
from .models import (
|
||||
DockerContainer, DockerDiskUsage, DockerEvent, DockerImage,
|
||||
DockerContainer, DockerDiskUsage, DockerEvent, DockerImage, DockerMetric,
|
||||
DockerSwarmNode, DockerSwarmService,
|
||||
)
|
||||
|
||||
docker_bp = Blueprint("docker", __name__, template_folder="templates")
|
||||
|
||||
|
||||
def _error(status: int, code: str, detail: str | None = None):
|
||||
body: dict = {"ok": False, "error": code}
|
||||
if detail:
|
||||
body["detail"] = detail
|
||||
return jsonify(body), status
|
||||
|
||||
|
||||
def _human_bytes(n: int | None) -> str:
|
||||
"""Compact binary size string (e.g. '1.4 GiB', '512 MiB', '0 B')."""
|
||||
if n is None:
|
||||
@@ -85,6 +74,38 @@ def _sparkline(values: list[float], width: int = 80, height: int = 20) -> str:
|
||||
)
|
||||
|
||||
|
||||
def _bucket(values: list[float], target: int = 40) -> list[float]:
|
||||
"""Bucket-average a series down to ~target points (DB-agnostic, in Python).
|
||||
|
||||
Replaces the old SQL strftime() bucketing, which was SQLite-only and broke
|
||||
on Postgres. Agent cadence is dense, so a multi-hour window is hundreds of
|
||||
rows — averaging into a readable point count keeps the sparkline's shape.
|
||||
"""
|
||||
n = len(values)
|
||||
if n <= target:
|
||||
return values
|
||||
size = (n + target - 1) // target
|
||||
out: list[float] = []
|
||||
for i in range(0, n, size):
|
||||
chunk = values[i:i + size]
|
||||
out.append(sum(chunk) / len(chunk))
|
||||
return out
|
||||
|
||||
|
||||
async def _container_history(db, host_id: str, name: str, since) -> tuple[list, list]:
|
||||
"""Return (cpu_series, mem_series) sparkline-ready for one host's container."""
|
||||
rows = (await db.execute(
|
||||
select(DockerMetric.cpu_pct, DockerMetric.mem_pct)
|
||||
.where(DockerMetric.host_id == host_id)
|
||||
.where(DockerMetric.container_name == name)
|
||||
.where(DockerMetric.scraped_at >= since)
|
||||
.order_by(DockerMetric.scraped_at)
|
||||
)).all()
|
||||
cpu = _bucket([r.cpu_pct or 0.0 for r in rows])
|
||||
mem = _bucket([r.mem_pct or 0.0 for r in rows])
|
||||
return cpu, mem
|
||||
|
||||
|
||||
@docker_bp.get("/")
|
||||
@require_role(UserRole.viewer)
|
||||
async def index():
|
||||
@@ -106,6 +127,15 @@ async def index():
|
||||
)
|
||||
|
||||
|
||||
async def _host_map(db, host_ids: set[str]) -> dict[str, Host]:
|
||||
if not host_ids:
|
||||
return {}
|
||||
return {
|
||||
h.id: h for h in (await db.execute(
|
||||
select(Host).where(Host.id.in_(host_ids)))).scalars().all()
|
||||
}
|
||||
|
||||
|
||||
@docker_bp.get("/rows")
|
||||
@require_role(UserRole.viewer)
|
||||
async def rows():
|
||||
@@ -114,10 +144,15 @@ async def rows():
|
||||
since, range_key = parse_range(request.args.get("range"))
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
containers = await docker_svc.list_containers(db)
|
||||
containers = list((await db.execute(
|
||||
select(DockerContainer).order_by(
|
||||
(DockerContainer.status == "running").desc(),
|
||||
DockerContainer.name,
|
||||
)
|
||||
)).scalars())
|
||||
services = list((await db.execute(select(DockerSwarmService))).scalars())
|
||||
nodes = list((await db.execute(select(DockerSwarmNode))).scalars())
|
||||
hosts = await docker_svc.host_map(
|
||||
hosts = await _host_map(
|
||||
db,
|
||||
{c.host_id for c in containers}
|
||||
| {n.host_id for n in nodes} | {s.host_id for s in services},
|
||||
@@ -128,9 +163,7 @@ async def rows():
|
||||
# from the managers' placement). Non-swarm containers keep the host view.
|
||||
node_hostname = {n.node_id: n.hostname for n in nodes if n.hostname}
|
||||
host_name = {h.id: h.name for h in hosts.values()}
|
||||
swarm_view = build_swarm_services(
|
||||
containers, services, node_hostname, host_name,
|
||||
crash_loop_deaths=docker_svc.crash_loop_deaths(current_app.config))
|
||||
swarm_view = build_swarm_services(containers, services, node_hostname, host_name)
|
||||
|
||||
standalone = [c for c in containers if not c.service_name]
|
||||
groups: dict[str, dict] = {}
|
||||
@@ -144,7 +177,7 @@ async def rows():
|
||||
"host_name": host.name if host else c.host_id,
|
||||
"containers": [], "running": 0, "stopped": 0,
|
||||
}
|
||||
cpu_hist, mem_hist = await docker_svc.container_history(db, c.host_id, c.name, since)
|
||||
cpu_hist, mem_hist = await _container_history(db, c.host_id, c.name, since)
|
||||
g["containers"].append({
|
||||
"container": c,
|
||||
"ports": json.loads(c.ports_json) if c.ports_json else [],
|
||||
@@ -206,14 +239,14 @@ async def widget():
|
||||
)).scalars()))
|
||||
services = list((await db.execute(select(DockerSwarmService))).scalars())
|
||||
nodes = list((await db.execute(select(DockerSwarmNode))).scalars())
|
||||
# Recent failures are more actionable than a static "stopped" tally. Read
|
||||
# Recent failures are more actionable than a static "stopped" tally. Count
|
||||
# DISTINCT container names so the two managers' duplicate die/oom events
|
||||
# don't double it; swarm tasks are counted on their service below.
|
||||
dead_names = list((await db.execute(
|
||||
select(func.distinct(DockerEvent.container_name))
|
||||
# don't double it.
|
||||
failed_24h = (await db.execute(
|
||||
select(func.count(func.distinct(DockerEvent.container_name)))
|
||||
.where(DockerEvent.event.in_(("die", "oom")), DockerEvent.at >= since_24h)
|
||||
)).scalars())
|
||||
hosts = await docker_svc.host_map(
|
||||
)).scalar() or 0
|
||||
hosts = await _host_map(
|
||||
db,
|
||||
{c.host_id for c in all_containers}
|
||||
| {n.host_id for n in nodes} | {s.host_id for s in services},
|
||||
@@ -223,10 +256,7 @@ async def widget():
|
||||
# host); non-swarm containers stay grouped by host.
|
||||
node_hostname = {n.node_id: n.hostname for n in nodes if n.hostname}
|
||||
host_name = {h.id: h.name for h in hosts.values()}
|
||||
swarm_services = build_swarm_services(
|
||||
all_containers, services, node_hostname, host_name,
|
||||
crash_loop_deaths=docker_svc.crash_loop_deaths(current_app.config))["services"]
|
||||
failed_24h = docker_svc.failed_24h(dead_names, swarm_services)
|
||||
swarm_services = build_swarm_services(all_containers, services, node_hostname, host_name)["services"]
|
||||
|
||||
standalone = [c for c in all_containers if not c.service_name]
|
||||
running = [c for c in standalone if c.status == "running"]
|
||||
@@ -271,7 +301,7 @@ async def widget_resources():
|
||||
.where(DockerContainer.status == "running")
|
||||
.order_by(DockerContainer.cpu_pct.desc().nullslast())
|
||||
)).scalars()))[:limit]
|
||||
hosts = await docker_svc.host_map(db, {c.host_id for c in containers})
|
||||
hosts = await _host_map(db, {c.host_id for c in containers})
|
||||
|
||||
rows_data = [
|
||||
{"c": c, "host_name": hosts[c.host_id].name if c.host_id in hosts else c.host_id}
|
||||
@@ -294,7 +324,14 @@ async def host_panel(host_id: str):
|
||||
don't carry an empty card.
|
||||
"""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
containers = await docker_svc.list_containers(db, host_id)
|
||||
containers = list((await db.execute(
|
||||
select(DockerContainer)
|
||||
.where(DockerContainer.host_id == host_id)
|
||||
.order_by(
|
||||
(DockerContainer.status == "running").desc(),
|
||||
DockerContainer.name,
|
||||
)
|
||||
)).scalars())
|
||||
|
||||
if not containers:
|
||||
return ""
|
||||
@@ -324,7 +361,15 @@ async def container_detail(host_id: str, name: str):
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
c = await db.get(DockerContainer, (host_id, name))
|
||||
host = await db.get(Host, host_id)
|
||||
events = await docker_svc.container_events(db, host_id, name) if c is not None else []
|
||||
events = []
|
||||
if c is not None:
|
||||
events = list((await db.execute(
|
||||
select(DockerEvent)
|
||||
.where(DockerEvent.host_id == host_id)
|
||||
.where(DockerEvent.container_name == name)
|
||||
.order_by(DockerEvent.at.desc())
|
||||
.limit(50)
|
||||
)).scalars())
|
||||
|
||||
if c is None:
|
||||
return await render_template(
|
||||
@@ -372,7 +417,7 @@ async def swarm():
|
||||
select(DockerSwarmNode).order_by(
|
||||
DockerSwarmNode.role, DockerSwarmNode.hostname)
|
||||
)).scalars())
|
||||
hosts = await docker_svc.host_map(db, {s.host_id for s in services} | {n.host_id for n in nodes})
|
||||
hosts = await _host_map(db, {s.host_id for s in services} | {n.host_id for n in nodes})
|
||||
|
||||
# ── Group reporting managers into swarms by shared node membership ──────────
|
||||
manager_ids = {s.host_id for s in services} | {n.host_id for n in nodes}
|
||||
@@ -418,8 +463,6 @@ async def swarm():
|
||||
node_by_id[n.node_id] = n
|
||||
|
||||
node_names = {nid: (nrow.hostname or nid) for nid, nrow in node_by_id.items()}
|
||||
now = datetime.now(timezone.utc)
|
||||
deaths = docker_svc.crash_loop_deaths(current_app.config)
|
||||
svc_dicts = []
|
||||
for s in sorted(svc_by_name.values(), key=lambda s: s.service_name):
|
||||
placement = []
|
||||
@@ -435,10 +478,7 @@ async def swarm():
|
||||
"healthy": s.running >= s.desired and s.desired > 0,
|
||||
"degraded": 0 < s.running < s.desired,
|
||||
"down": s.running == 0 and s.desired > 0,
|
||||
"state": ("healthy" if s.running >= s.desired and s.desired > 0
|
||||
else "degraded" if s.running > 0 else "down"),
|
||||
"placement": placement,
|
||||
**task_summary(parse_tasks(s), now, deaths, node_names),
|
||||
})
|
||||
node_rows = sorted(
|
||||
node_by_id.values(), key=lambda n: (n.role, (n.hostname or n.node_id).lower())
|
||||
@@ -459,13 +499,9 @@ async def swarm():
|
||||
async def disk():
|
||||
"""Image/disk usage page: reclaimable space, per-image sizes, stopped count.
|
||||
|
||||
Admins can reclaim space per host via the prune buttons, which fire the
|
||||
audited bundled prune playbook through Ansible (m78) — the collection agent
|
||||
itself stays read-only.
|
||||
Read-only — prune actions are deferred to the cleanup-actions milestone, so
|
||||
this surfaces the numbers and notes where reclaim lives.
|
||||
"""
|
||||
from steward.core.capabilities import has_capability
|
||||
from steward.models.ansible_inventory import AnsibleTarget
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
summaries = list((await db.execute(select(DockerDiskUsage))).scalars())
|
||||
images = list((await db.execute(
|
||||
@@ -479,17 +515,7 @@ async def disk():
|
||||
stopped_by_host: dict[str, int] = {}
|
||||
for hid in stopped_rows:
|
||||
stopped_by_host[hid] = stopped_by_host.get(hid, 0) + 1
|
||||
hosts = await docker_svc.host_map(db, {s.host_id for s in summaries})
|
||||
# Which hosts have a linked Ansible target — gates the prune buttons
|
||||
# (a target is required to route the playbook run to that host).
|
||||
host_ids = {s.host_id for s in summaries}
|
||||
target_by_host: dict[str, str] = {}
|
||||
if host_ids:
|
||||
for hid, tid in (await db.execute(
|
||||
select(AnsibleTarget.host_id, AnsibleTarget.id)
|
||||
.where(AnsibleTarget.host_id.in_(host_ids))
|
||||
)).all():
|
||||
target_by_host[hid] = tid
|
||||
hosts = await _host_map(db, {s.host_id for s in summaries})
|
||||
|
||||
images_by_host: dict[str, list] = {}
|
||||
for im in images:
|
||||
@@ -516,60 +542,11 @@ async def disk():
|
||||
},
|
||||
"stopped": stopped_by_host.get(s.host_id, 0),
|
||||
"images": images_by_host.get(s.host_id, []),
|
||||
"has_target": s.host_id in target_by_host,
|
||||
}
|
||||
for s in summaries
|
||||
]
|
||||
host_groups.sort(key=lambda g: g["host_name"].lower())
|
||||
return await render_template(
|
||||
"docker/disk.html", host_groups=host_groups,
|
||||
ansible_available=has_capability("ansible.run_playbook"),
|
||||
)
|
||||
|
||||
|
||||
@docker_bp.post("/disk/<host_id>/prune")
|
||||
@require_role(UserRole.admin)
|
||||
async def disk_prune(host_id: str):
|
||||
"""Reclaim Docker disk on a host by firing the bundled prune playbook via
|
||||
Ansible (audited, admin-gated) — the collection agent stays read-only (m78).
|
||||
`target` selects the scope: containers | images | system.
|
||||
"""
|
||||
from steward.core.capabilities import has_capability, invoke_capability
|
||||
from steward.ansible.sources import BUILTIN_SOURCE_NAME
|
||||
from steward.models.ansible_inventory import AnsibleTarget
|
||||
|
||||
if not has_capability("ansible.run_playbook"):
|
||||
return _error(400, "ansible_unavailable", "Ansible is not available")
|
||||
|
||||
form = await request.form
|
||||
prune_target = (form.get("target", "") or "").strip()
|
||||
if prune_target not in docker_svc.PRUNE_TARGETS:
|
||||
return _error(400, "bad_target", "Unknown prune target")
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
target = (await db.execute(
|
||||
select(AnsibleTarget).where(AnsibleTarget.host_id == host_id)
|
||||
)).scalar_one_or_none()
|
||||
if target is None:
|
||||
return _error(400, "no_target",
|
||||
"Link an Ansible target to this host before pruning")
|
||||
|
||||
# extra_vars_map outranks the playbook's own `vars:` defaults.
|
||||
extra_vars = docker_svc.prune_extra_vars(prune_target)
|
||||
|
||||
actor_role = UserRole(session.get("user_role", "viewer"))
|
||||
run, _source, err = await invoke_capability(
|
||||
"ansible.run_playbook", actor_role,
|
||||
current_app._get_current_object(), # type: ignore[attr-defined]
|
||||
source_name=BUILTIN_SOURCE_NAME,
|
||||
playbook_path="maintenance/docker_prune.yml",
|
||||
inventory_scope=f"steward:target:{target.id}",
|
||||
params={"extra_vars_map": extra_vars},
|
||||
triggered_by=session.get("user_id"),
|
||||
)
|
||||
if err:
|
||||
return _error(400, "prune_failed", err)
|
||||
return redirect(url_for("ansible.run_detail", run_id=run.id))
|
||||
return await render_template("docker/disk.html", host_groups=host_groups)
|
||||
|
||||
|
||||
@docker_bp.get("/container/<host_id>/<name>/history")
|
||||
@@ -578,7 +555,7 @@ async def container_history(host_id: str, name: str):
|
||||
"""HTMX fragment: CPU + memory sparklines for the selected time range."""
|
||||
since, range_key = parse_range(request.args.get("range"))
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
cpu_hist, mem_hist = await docker_svc.container_history(db, host_id, name, since)
|
||||
cpu_hist, mem_hist = await _container_history(db, host_id, name, since)
|
||||
return await render_template(
|
||||
"docker/_container_history.html",
|
||||
sparkline_cpu=_sparkline(cpu_hist, width=320, height=48),
|
||||
@@ -586,36 +563,3 @@ async def container_history(host_id: str, name: str):
|
||||
have_data=len(cpu_hist) >= 2,
|
||||
range_key=range_key,
|
||||
)
|
||||
|
||||
|
||||
# Newest lines returned per fetch — the retained window is bounded by the ring,
|
||||
# but a container can still hold thousands of lines; cap what one fetch renders.
|
||||
_LOG_TAIL_DEFAULT = 500
|
||||
|
||||
|
||||
@docker_bp.get("/container/<host_id>/<name>/logs")
|
||||
@require_role(UserRole.viewer)
|
||||
async def container_logs(host_id: str, name: str):
|
||||
"""Full log-viewer page for one container (lines load via an HTMX fragment)."""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
host = await db.get(Host, host_id)
|
||||
return await render_template(
|
||||
"docker/container_logs.html", host=host, host_id=host_id, name=name,
|
||||
)
|
||||
|
||||
|
||||
@docker_bp.get("/container/<host_id>/<name>/logs/lines")
|
||||
@require_role(UserRole.viewer)
|
||||
async def container_logs_lines(host_id: str, name: str):
|
||||
"""HTMX fragment: the recent retained log lines, stream + text filtered.
|
||||
Polled every few seconds by the viewer for near-live follow."""
|
||||
stream = request.args.get("stream", "all")
|
||||
query = (request.args.get("q") or "").strip()
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
rows = await docker_svc.query_logs(db, host_id, name, stream, query, _LOG_TAIL_DEFAULT)
|
||||
return await render_template(
|
||||
"docker/_container_logs_lines.html",
|
||||
lines=[{"ts": r.ts, "stream": r.stream, "line": r.line} for r in rows],
|
||||
name=name, tail=_LOG_TAIL_DEFAULT,
|
||||
filtered=bool(query) or stream in ("stdout", "stderr"),
|
||||
)
|
||||
|
||||
@@ -1,280 +0,0 @@
|
||||
"""Docker queries shared by the plugin's pages and its MCP tools.
|
||||
|
||||
Each function takes an open session and returns rows or plain data, with no
|
||||
Quart request context, so the routes and the /mcp tools (which run outside
|
||||
Quart's pipeline) call the same code.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from steward.core.time_range import bucket_seconds
|
||||
from steward.models.hosts import Host
|
||||
|
||||
from .models import DockerContainer, DockerEvent, DockerLog, DockerMetric, DockerMetricHourly
|
||||
|
||||
|
||||
CRASH_LOOP_DEATHS = 3 # default; Settings → Thresholds sets it
|
||||
|
||||
|
||||
def crash_loop_deaths(config) -> int:
|
||||
"""Deaths within an hour that make a container or a swarm service a crash
|
||||
loop, from Settings → Thresholds (app.config["THRESHOLDS"])."""
|
||||
th = (config.get("THRESHOLDS") or {}).get("crash_loop") or {}
|
||||
return int(th.get("warn") or CRASH_LOOP_DEATHS)
|
||||
|
||||
|
||||
def failed_24h(dead_names, swarm_services) -> int:
|
||||
"""Failures in the last 24h for the dashboard: distinct plain containers
|
||||
that died or were OOM-killed, plus each swarm service's failed tasks.
|
||||
|
||||
A swarm task's container dies whenever Swarm replaces it (a rolling update
|
||||
kills it with 137), so its die event is not a failure. Services with
|
||||
Swarm's task records count their failed tasks instead; a service without
|
||||
them (an agent older than 1.8.0) falls back to its tasks' die events.
|
||||
"""
|
||||
by_prefix = {s["name"] + ".": s for s in swarm_services}
|
||||
plain, fallback = 0, set()
|
||||
for name in dead_names:
|
||||
svc = next((s for p, s in by_prefix.items() if name.startswith(p)), None)
|
||||
if svc is None:
|
||||
plain += 1
|
||||
elif not svc["tasks_known"]:
|
||||
fallback.add(name)
|
||||
tasks = sum(s["failures_24h"] or 0 for s in swarm_services if s["tasks_known"])
|
||||
return plain + len(fallback) + tasks
|
||||
|
||||
|
||||
def bucket(values: list[float], target: int = 40) -> list[float]:
|
||||
"""Bucket-average a series down to ~target points (DB-agnostic, in Python).
|
||||
|
||||
Replaces the old SQL strftime() bucketing, which was SQLite-only and broke
|
||||
on Postgres. Agent cadence is dense, so a multi-hour window is hundreds of
|
||||
rows — averaging into a readable point count keeps the sparkline's shape.
|
||||
"""
|
||||
n = len(values)
|
||||
if n <= target:
|
||||
return values
|
||||
size = (n + target - 1) // target
|
||||
out: list[float] = []
|
||||
for i in range(0, n, size):
|
||||
chunk = values[i:i + size]
|
||||
out.append(sum(chunk) / len(chunk))
|
||||
return out
|
||||
|
||||
|
||||
async def container_history(db, host_id: str, name: str, since) -> tuple[list, list]:
|
||||
"""Return (cpu_series, mem_series) sparkline-ready for one host's container."""
|
||||
rows = (await db.execute(
|
||||
select(DockerMetric.cpu_pct, DockerMetric.mem_pct)
|
||||
.where(DockerMetric.host_id == host_id)
|
||||
.where(DockerMetric.container_name == name)
|
||||
.where(DockerMetric.scraped_at >= since)
|
||||
.order_by(DockerMetric.scraped_at)
|
||||
)).all()
|
||||
cpu = bucket([r.cpu_pct or 0.0 for r in rows])
|
||||
mem = bucket([r.mem_pct or 0.0 for r in rows])
|
||||
return cpu, mem
|
||||
|
||||
|
||||
async def host_map(db, host_ids: set[str]) -> dict[str, Host]:
|
||||
if not host_ids:
|
||||
return {}
|
||||
return {
|
||||
h.id: h for h in (await db.execute(
|
||||
select(Host).where(Host.id.in_(host_ids)))).scalars().all()
|
||||
}
|
||||
|
||||
|
||||
def escape_like(text: str) -> str:
|
||||
"""Make `text` match literally inside a LIKE/ILIKE pattern (escape char \\)."""
|
||||
return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
async def list_containers(db, host_id: str | None = None) -> list[DockerContainer]:
|
||||
"""Containers, running first then by name; one host's when host_id is given."""
|
||||
stmt = select(DockerContainer)
|
||||
if host_id is not None:
|
||||
stmt = stmt.where(DockerContainer.host_id == host_id)
|
||||
stmt = stmt.order_by((DockerContainer.status == "running").desc(), DockerContainer.name)
|
||||
return list((await db.execute(stmt)).scalars())
|
||||
|
||||
|
||||
async def container_events(db, host_id: str, name: str, limit: int = 50) -> list[DockerEvent]:
|
||||
"""A container's lifecycle events (start/stop/die/oom/health), newest first."""
|
||||
return list((await db.execute(
|
||||
select(DockerEvent)
|
||||
.where(DockerEvent.host_id == host_id)
|
||||
.where(DockerEvent.container_name == name)
|
||||
.order_by(DockerEvent.at.desc())
|
||||
.limit(limit)
|
||||
)).scalars())
|
||||
|
||||
|
||||
async def query_logs(db, host_id: str, name: str, stream: str, query: str,
|
||||
limit: int) -> list:
|
||||
"""The recent retained lines for one container, newest-first, optionally
|
||||
filtered by stream and a case-insensitive substring.
|
||||
|
||||
Newest-first is deliberate: the viewer replaces the list on each poll, which
|
||||
resets scroll to the top — so the latest lines stay visible without any
|
||||
scroll handling, and older lines are a scroll away.
|
||||
"""
|
||||
stmt = (select(DockerLog.ts, DockerLog.stream, DockerLog.line)
|
||||
.where(DockerLog.host_id == host_id)
|
||||
.where(DockerLog.container_name == name))
|
||||
if stream in ("stdout", "stderr"):
|
||||
stmt = stmt.where(DockerLog.stream == stream)
|
||||
if query:
|
||||
stmt = stmt.where(DockerLog.line.ilike(f"%{escape_like(query)}%", escape="\\"))
|
||||
stmt = stmt.order_by(DockerLog.ts.desc(), DockerLog.id.desc()).limit(limit)
|
||||
return (await db.execute(stmt)).all()
|
||||
|
||||
|
||||
CONTAINER_SERIES = ("cpu_pct", "mem_pct", "mem_usage_bytes")
|
||||
|
||||
|
||||
async def container_points(db, host_id: str, name: str, metric: str, since,
|
||||
target: int = 120) -> list[tuple]:
|
||||
"""(time, average) points of one container metric since `since`, oldest first.
|
||||
|
||||
Raw samples are bucket-averaged in SQL; the part of the range older than
|
||||
the oldest raw sample (already rolled up and pruned by retention) comes
|
||||
from the hourly table.
|
||||
"""
|
||||
if metric not in CONTAINER_SERIES:
|
||||
raise ValueError(f"metric must be one of {', '.join(CONTAINER_SERIES)}")
|
||||
raw_col = getattr(DockerMetric, metric)
|
||||
width_s = bucket_seconds(since, target)
|
||||
rbucket = func.date_bin(
|
||||
func.make_interval(0, 0, 0, 0, 0, 0, width_s), DockerMetric.scraped_at, func.to_timestamp(0),
|
||||
).label("bucket")
|
||||
raw = (await db.execute(
|
||||
select(rbucket, func.avg(raw_col))
|
||||
.where(DockerMetric.host_id == host_id, DockerMetric.container_name == name,
|
||||
DockerMetric.scraped_at >= since)
|
||||
.group_by(rbucket).order_by(rbucket)
|
||||
)).all()
|
||||
oldest_raw = raw[0][0] if raw else None
|
||||
hourly_stmt = (
|
||||
select(DockerMetricHourly.bucket, getattr(DockerMetricHourly, metric))
|
||||
.where(DockerMetricHourly.host_id == host_id,
|
||||
DockerMetricHourly.container_name == name,
|
||||
DockerMetricHourly.bucket >= since)
|
||||
.order_by(DockerMetricHourly.bucket)
|
||||
)
|
||||
if oldest_raw is not None:
|
||||
hourly_stmt = hourly_stmt.where(DockerMetricHourly.bucket < oldest_raw)
|
||||
hourly = (await db.execute(hourly_stmt)).all()
|
||||
return [(t, float(v)) for t, v in hourly] + [(t, float(v)) for t, v in raw]
|
||||
|
||||
|
||||
async def search_logs(db, *, since, host_id: str | None = None, name: str | None = None,
|
||||
text: str = "", stream: str = "", before=None, limit: int = 50) -> list:
|
||||
"""Retained log lines matching a filter, newest first (limit + 1 fetched by callers
|
||||
that need to know whether more exist)."""
|
||||
stmt = (select(DockerLog.id, DockerLog.host_id, DockerLog.container_name,
|
||||
DockerLog.ts, DockerLog.stream, DockerLog.line)
|
||||
.where(DockerLog.ts >= since))
|
||||
if host_id:
|
||||
stmt = stmt.where(DockerLog.host_id == host_id)
|
||||
if name:
|
||||
stmt = stmt.where(DockerLog.container_name == name)
|
||||
if stream in ("stdout", "stderr"):
|
||||
stmt = stmt.where(DockerLog.stream == stream)
|
||||
if text:
|
||||
stmt = stmt.where(DockerLog.line.ilike(f"%{escape_like(text)}%", escape="\\"))
|
||||
if before is not None:
|
||||
stmt = stmt.where(DockerLog.ts < before)
|
||||
stmt = stmt.order_by(DockerLog.ts.desc(), DockerLog.id.desc()).limit(limit)
|
||||
return list((await db.execute(stmt)).all())
|
||||
|
||||
|
||||
async def log_context(db, host_id: str, name: str, ts, n: int) -> tuple[list, list]:
|
||||
"""Up to n lines either side of `ts` in one container's log, each oldest first."""
|
||||
cols = (DockerLog.id, DockerLog.ts, DockerLog.stream, DockerLog.line)
|
||||
base = select(*cols).where(DockerLog.host_id == host_id, DockerLog.container_name == name)
|
||||
before = (await db.execute(
|
||||
base.where(DockerLog.ts < ts).order_by(DockerLog.ts.desc(), DockerLog.id.desc()).limit(n)
|
||||
)).all()
|
||||
after = (await db.execute(
|
||||
base.where(DockerLog.ts > ts).order_by(DockerLog.ts, DockerLog.id).limit(n)
|
||||
)).all()
|
||||
return list(reversed(before)), list(after)
|
||||
|
||||
|
||||
async def recent_failures_by_container(db, since) -> dict[tuple[str, str], list]:
|
||||
"""{(host_id, container_name): [die/oom events since `since`, newest first]}."""
|
||||
rows = (await db.execute(
|
||||
select(DockerEvent)
|
||||
.where(DockerEvent.event.in_(("die", "oom")), DockerEvent.at >= since)
|
||||
.order_by(DockerEvent.at.desc())
|
||||
)).scalars()
|
||||
out: dict[tuple[str, str], list] = {}
|
||||
for e in rows:
|
||||
out.setdefault((e.host_id, e.container_name), []).append(e)
|
||||
return out
|
||||
|
||||
|
||||
async def retained_names(db, want: str, *, exact: bool, host_id: str | None = None,
|
||||
limit: int = 20) -> list[tuple[str, str, datetime]]:
|
||||
"""(host_id, container_name, last seen) of containers Steward still holds
|
||||
logs or events for, newest first — including ones Docker has removed.
|
||||
|
||||
exact: the name itself, from both logs and events. Otherwise names that
|
||||
contain `want` (case-insensitive), from events only: a substring match
|
||||
over every retained log line's name would scan the log table, and a
|
||||
container Steward watched die has a die or stop event.
|
||||
"""
|
||||
seen: dict[tuple[str, str], datetime] = {}
|
||||
sources = ((DockerLog, DockerLog.ts), (DockerEvent, DockerEvent.at)) if exact \
|
||||
else ((DockerEvent, DockerEvent.at),)
|
||||
for model, at in sources:
|
||||
match = (model.container_name == want if exact
|
||||
else model.container_name.ilike(f"%{escape_like(want)}%", escape="\\"))
|
||||
stmt = (select(model.host_id, model.container_name, func.max(at))
|
||||
.where(match).group_by(model.host_id, model.container_name))
|
||||
if host_id:
|
||||
stmt = stmt.where(model.host_id == host_id)
|
||||
for hid, name, last in (await db.execute(stmt)).all():
|
||||
key = (hid, name)
|
||||
if key not in seen or last > seen[key]:
|
||||
seen[key] = last
|
||||
rows = sorted(((h, n, t) for (h, n), t in seen.items()), key=lambda r: r[2], reverse=True)
|
||||
return rows[:limit]
|
||||
|
||||
|
||||
async def unhealthy_since(db, keys) -> dict[tuple[str, str], datetime]:
|
||||
"""{(host_id, container_name): when it last turned unhealthy} for the given
|
||||
containers, from their health_change events; absent when none is retained."""
|
||||
keys = set(keys)
|
||||
if not keys:
|
||||
return {}
|
||||
rows = (await db.execute(
|
||||
select(DockerEvent)
|
||||
.where(DockerEvent.event == "health_change",
|
||||
DockerEvent.detail.like("%→unhealthy"),
|
||||
DockerEvent.container_name.in_({name for _, name in keys}))
|
||||
.order_by(DockerEvent.at.desc())
|
||||
)).scalars()
|
||||
out: dict[tuple[str, str], datetime] = {}
|
||||
for e in rows:
|
||||
key = (e.host_id, e.container_name)
|
||||
if key in keys and key not in out:
|
||||
out[key] = e.at
|
||||
return out
|
||||
|
||||
|
||||
PRUNE_TARGETS = ("containers", "images", "system")
|
||||
|
||||
|
||||
def prune_extra_vars(prune_target: str) -> dict:
|
||||
"""Extra-vars for the bundled docker_prune playbook. "Prune unused images"
|
||||
means ALL unused (docker image prune -a), not just dangling; the full system
|
||||
prune stays conservative (-f, no -a) per the operator's choice (m78)."""
|
||||
extra_vars: dict = {"prune_target": prune_target}
|
||||
if prune_target == "images":
|
||||
extra_vars["prune_all_images"] = True
|
||||
return extra_vars
|
||||
+14
-226
@@ -11,28 +11,12 @@ For every swarm service we list the real replicas we have detail for, then add
|
||||
"ghost" replicas for the remaining placement count on each node (agent-less
|
||||
nodes). Non-swarm containers pass through grouped by host, unchanged.
|
||||
|
||||
A third source, DockerSwarmService.tasks_json (agent ≥ 1.8.0), is Swarm's own
|
||||
record of each service's newest tasks. It decides which task containers are
|
||||
*superseded* — replaced by Swarm, so neither a replica nor a crash — and gives
|
||||
each service its failures across task retries and the error of its last
|
||||
failed task. Swarm restarts a task by creating a new one (a new container), so
|
||||
failures can only be counted per service, never per container.
|
||||
|
||||
Kept model-free (no ORM imports) so it's unit-testable and safe to import in
|
||||
tests without the plugin-loader "table already defined" gotcha.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
# Task states that mean the task ended in error: `failed` (its container exited
|
||||
# non-zero or never started) and `rejected` (Swarm could not start it at all,
|
||||
# e.g. no such image, a bad mount).
|
||||
FAILED_TASK_STATES = ("failed", "rejected")
|
||||
# Desired states of a task Swarm still means to run. Anything else (shutdown,
|
||||
# remove, complete, orphaned) is a task Swarm has finished with.
|
||||
_CURRENT_DESIRED = ("running", "ready", "")
|
||||
|
||||
|
||||
def _service_state(running: int, desired: int) -> str:
|
||||
@@ -52,220 +36,30 @@ def _placement(service) -> list[dict]:
|
||||
return [p for p in data if isinstance(p, dict)]
|
||||
|
||||
|
||||
def parse_tasks(service) -> list[dict]:
|
||||
"""A service row's task records, newest first; [] when its agent sent none."""
|
||||
raw = getattr(service, "tasks_json", None) or "[]"
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
return []
|
||||
return [t for t in data if isinstance(t, dict)]
|
||||
def build_swarm_services(containers, services, node_hostname: dict, host_name: dict) -> dict:
|
||||
"""Return {"services": [...], "standalone": [...]}.
|
||||
|
||||
|
||||
def task_time(task: dict) -> datetime | None:
|
||||
"""A task's status timestamp. Docker writes nanoseconds; Python ≥ 3.11 keeps
|
||||
the first six digits."""
|
||||
raw = task.get("at") or ""
|
||||
try:
|
||||
at = datetime.fromisoformat(raw.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return at if at.tzinfo else at.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def task_failures(tasks: list[dict], since: datetime) -> int:
|
||||
"""How many tasks ended in error at or after `since`."""
|
||||
n = 0
|
||||
for t in tasks:
|
||||
at = task_time(t)
|
||||
if t.get("state") in FAILED_TASK_STATES and at is not None and at >= since:
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def last_task_error(tasks: list[dict]) -> dict | None:
|
||||
"""Why the service's newest troubled task is in trouble, from Swarm itself.
|
||||
|
||||
The newest task that failed or was rejected, or that Swarm attached an error
|
||||
to while still trying (a pending task with "no suitable node", say). None
|
||||
when no retained task has a problem.
|
||||
"""
|
||||
for t in tasks: # newest first
|
||||
if t.get("state") in FAILED_TASK_STATES or t.get("error"):
|
||||
return {"state": t.get("state"), "error": t.get("error") or None,
|
||||
"message": t.get("message") or None, "exit_code": t.get("exit_code"),
|
||||
"node_id": t.get("node_id") or None, "at": t.get("at") or None}
|
||||
return None
|
||||
|
||||
|
||||
def task_summary(tasks: list[dict], now: datetime, crash_loop_deaths: int,
|
||||
node_hostname: dict) -> dict:
|
||||
"""What a service's task records say: tasks_known, failures_1h,
|
||||
failures_24h (None without records), crash_looping, and last_error with
|
||||
the node's hostname as `node`."""
|
||||
known = bool(tasks)
|
||||
failures_1h = task_failures(tasks, now - timedelta(hours=1)) if known else None
|
||||
last_error = last_task_error(tasks)
|
||||
if last_error is not None:
|
||||
nid = last_error["node_id"]
|
||||
last_error["node"] = node_hostname.get(nid) or (nid[:12] if nid else None)
|
||||
at = task_time(last_error)
|
||||
last_error["age_s"] = max(0, int((now - at).total_seconds())) if at else None
|
||||
return {
|
||||
"tasks_known": known,
|
||||
"failures_1h": failures_1h,
|
||||
"failures_24h": task_failures(tasks, now - timedelta(hours=24)) if known else None,
|
||||
"crash_looping": bool(known and failures_1h >= crash_loop_deaths),
|
||||
"last_error": last_error,
|
||||
}
|
||||
|
||||
|
||||
def _oldest_failure(tasks: list[dict], since: datetime) -> datetime | None:
|
||||
times = [at for t in tasks if t.get("state") in FAILED_TASK_STATES
|
||||
and (at := task_time(t)) is not None and at >= since]
|
||||
return min(times, default=None)
|
||||
|
||||
|
||||
def trouble_since(tasks: list[dict], state: str, crash_looping: bool,
|
||||
now: datetime) -> datetime | None:
|
||||
"""When a service's current trouble began, as far as its retained tasks
|
||||
tell; None when it has none, or the tasks cannot say.
|
||||
|
||||
A crash loop dates from its first failure in the last hour, and a healthy
|
||||
service that lost tasks from its first failure in the last 24h. A service
|
||||
short of replicas dates from its troubled slots (replicated: the slot;
|
||||
global: the node): in each slot whose newest task is not running, the
|
||||
oldest of its newest troubled tasks (failed, rejected, or carrying an
|
||||
error), walking back until a running task interrupts them. Swarm keeps few
|
||||
old tasks, so a long outage reads as the oldest one retained: a lower
|
||||
bound.
|
||||
"""
|
||||
if crash_looping:
|
||||
return _oldest_failure(tasks, now - timedelta(hours=1))
|
||||
if state == "healthy":
|
||||
return _oldest_failure(tasks, now - timedelta(hours=24))
|
||||
by_slot: dict = {}
|
||||
for t in tasks: # newest first, so each slot's list is too
|
||||
by_slot.setdefault(t.get("slot") or t.get("node_id"), []).append(t)
|
||||
began = []
|
||||
for slot_tasks in by_slot.values():
|
||||
oldest = None
|
||||
for t in slot_tasks:
|
||||
if t.get("state") == "running":
|
||||
break
|
||||
if t.get("state") in FAILED_TASK_STATES or t.get("error"):
|
||||
oldest = task_time(t) or oldest
|
||||
if oldest is not None:
|
||||
began.append(oldest)
|
||||
return min(began, default=None)
|
||||
|
||||
|
||||
def _slot_key(c) -> str:
|
||||
"""A task container's slot within its service: "<service>.<slot>.<task>" for
|
||||
replicated services, "<service>.<node id>.<task>" for global ones."""
|
||||
svc, name = c.service_name, c.name or ""
|
||||
rest = name[len(svc) + 1:] if name.startswith(svc + ".") else name
|
||||
return rest.split(".", 1)[0]
|
||||
|
||||
|
||||
def superseded_containers(swarm_cs, tasks_by_id: dict[str, dict]) -> set[tuple]:
|
||||
"""(host_id, name) of task containers Swarm has replaced.
|
||||
|
||||
With the task on record, its desired state decides: Swarm sets a replaced
|
||||
or failed task's desired state to shutdown. Without it (an older agent, or a
|
||||
task older than the retained history), a container that is not running is
|
||||
superseded when another task in the same slot is running or started later.
|
||||
"""
|
||||
out: set[tuple] = set()
|
||||
by_slot: dict[tuple, list] = {}
|
||||
for c in swarm_cs:
|
||||
by_slot.setdefault((c.service_name, _slot_key(c)), []).append(c)
|
||||
for c in swarm_cs:
|
||||
task = tasks_by_id.get(getattr(c, "task_id", None) or "")
|
||||
if task is not None:
|
||||
if task.get("desired_state") not in _CURRENT_DESIRED:
|
||||
out.add((c.host_id, c.name))
|
||||
continue
|
||||
if (getattr(c, "status", "") or "").lower() == "running":
|
||||
continue
|
||||
mine = getattr(c, "started_at", None)
|
||||
for other in by_slot[(c.service_name, _slot_key(c))]:
|
||||
if other is c:
|
||||
continue
|
||||
theirs = getattr(other, "started_at", None)
|
||||
if ((getattr(other, "status", "") or "").lower() == "running"
|
||||
or (theirs is not None and (mine is None or theirs > mine))):
|
||||
out.add((c.host_id, c.name))
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _newest_per_name(services) -> dict[str, object]:
|
||||
"""Every manager reports the same cluster-global service set; keep the most
|
||||
recently updated row per name, so a manager on an older agent (no tasks)
|
||||
doesn't shadow one that reports them."""
|
||||
by_name: dict[str, object] = {}
|
||||
for s in services:
|
||||
name = getattr(s, "service_name", None)
|
||||
if not name:
|
||||
continue
|
||||
kept = by_name.get(name)
|
||||
if kept is None:
|
||||
by_name[name] = s
|
||||
continue
|
||||
a, b = getattr(s, "updated_at", None), getattr(kept, "updated_at", None)
|
||||
if a is not None and (b is None or a > b):
|
||||
by_name[name] = s
|
||||
return by_name
|
||||
|
||||
|
||||
def replaced_tasks(containers, services) -> set[tuple]:
|
||||
"""(host_id, name) of the task containers Swarm has replaced, from the
|
||||
container rows and the managers' service rows. What the container views use
|
||||
to leave history out of their problem lists."""
|
||||
swarm_cs = [c for c in containers if getattr(c, "service_name", None)]
|
||||
tasks_by_id = {t["task_id"]: t for s in _newest_per_name(services).values()
|
||||
for t in parse_tasks(s) if t.get("task_id")}
|
||||
return superseded_containers(swarm_cs, tasks_by_id)
|
||||
|
||||
|
||||
def build_swarm_services(containers, services, node_hostname: dict, host_name: dict,
|
||||
*, now: datetime | None = None, crash_loop_deaths: int = 3) -> dict:
|
||||
"""Return {"services": [...], "standalone": [...], "superseded": {(host_id, name)}}.
|
||||
|
||||
services[i] = {name, mode, desired, running, state, replicas:[...],
|
||||
previous_tasks, tasks_known, failures_1h, failures_24h,
|
||||
crash_looping, last_error}
|
||||
services[i] = {name, mode, desired, running, state, replicas:[...]}
|
||||
real replica = {ghost:False, host, host_id, name, status, cpu_pct, mem_pct,
|
||||
health, restart_count} # current tasks only
|
||||
health, restart_count}
|
||||
ghost replica = {ghost:True, host, count} # placement with no local row
|
||||
previous_tasks = task containers Swarm replaced, left out of replicas
|
||||
failures_* = tasks that failed or were rejected in the window (None
|
||||
without task data); crash_looping = failures_1h ≥ the
|
||||
Settings → Thresholds crash-loop count
|
||||
last_error = last_task_error(), plus `node` (the node's hostname)
|
||||
trouble_since = trouble_since(): when the current trouble began, or None
|
||||
standalone[i] = {host_id, host, containers:[...]} # non-swarm, by host
|
||||
superseded = the task containers Swarm replaced, so callers can skip them
|
||||
"""
|
||||
now = now or datetime.now(timezone.utc)
|
||||
containers = list(containers)
|
||||
swarm_cs = [c for c in containers if getattr(c, "service_name", None)]
|
||||
standalone_cs = [c for c in containers if not getattr(c, "service_name", None)]
|
||||
|
||||
svc_by_name = _newest_per_name(services)
|
||||
tasks_by_svc = {name: parse_tasks(s) for name, s in svc_by_name.items()}
|
||||
tasks_by_id = {t["task_id"]: t for ts in tasks_by_svc.values() for t in ts
|
||||
if t.get("task_id")}
|
||||
superseded = superseded_containers(swarm_cs, tasks_by_id)
|
||||
# Dedup service rows by name (every manager reports the same cluster-global
|
||||
# set); keep the first — placement is identical across managers.
|
||||
svc_by_name: dict[str, object] = {}
|
||||
for s in services:
|
||||
name = getattr(s, "service_name", None)
|
||||
if name and name not in svc_by_name:
|
||||
svc_by_name[name] = s
|
||||
|
||||
# Current task containers grouped by service name; replaced ones are counted.
|
||||
# Real containers grouped by service name.
|
||||
cs_by_service: dict[str, list] = {}
|
||||
previous: dict[str, int] = {}
|
||||
for c in swarm_cs:
|
||||
if (c.host_id, c.name) in superseded:
|
||||
previous[c.service_name] = previous.get(c.service_name, 0) + 1
|
||||
continue
|
||||
cs_by_service.setdefault(c.service_name, []).append(c)
|
||||
|
||||
def _host_of(c):
|
||||
@@ -323,19 +117,13 @@ def build_swarm_services(containers, services, node_hostname: dict, host_name: d
|
||||
if desired is None:
|
||||
desired = len(replicas)
|
||||
|
||||
tasks = tasks_by_svc.get(name, [])
|
||||
state = _service_state(int(running or 0), int(desired or 0))
|
||||
summary = task_summary(tasks, now, crash_loop_deaths, node_hostname)
|
||||
out_services.append({
|
||||
"name": name,
|
||||
"mode": getattr(svc, "mode", "replicated") if svc is not None else "replicated",
|
||||
"running": running,
|
||||
"desired": desired,
|
||||
"state": state,
|
||||
"state": _service_state(int(running or 0), int(desired or 0)),
|
||||
"replicas": replicas,
|
||||
"previous_tasks": previous.get(name, 0),
|
||||
**summary,
|
||||
"trouble_since": trouble_since(tasks, state, summary["crash_looping"], now),
|
||||
})
|
||||
|
||||
# Non-swarm containers grouped by host, running-first order preserved.
|
||||
@@ -349,4 +137,4 @@ def build_swarm_services(containers, services, node_hostname: dict, host_name: d
|
||||
g["containers"].append(c)
|
||||
standalone = sorted(groups.values(), key=lambda g: g["host"].lower())
|
||||
|
||||
return {"services": out_services, "standalone": standalone, "superseded": superseded}
|
||||
return {"services": out_services, "standalone": standalone}
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
{# docker/_container_logs_lines.html — log-line fragment, HTMX-polled (m79).
|
||||
{{ l.line }} is auto-escaped by Jinja, so log content can't inject markup. #}
|
||||
{% if lines %}
|
||||
{% for l in lines %}
|
||||
<div style="display:flex;gap:0.6rem;white-space:pre-wrap;word-break:break-word;">
|
||||
<span style="color:var(--text-dim);flex-shrink:0;" title="{{ l.ts }}">{{ l.ts.strftime("%m-%d %H:%M:%S") }}</span>
|
||||
<span style="flex-shrink:0;width:3.2rem;color:{% if l.stream == 'stderr' %}var(--red){% else %}var(--text-muted){% endif %};">{{ l.stream }}</span>
|
||||
<span style="flex:1;min-width:0;{% if l.stream == 'stderr' %}color:var(--text);{% endif %}">{{ l.line }}</span>
|
||||
</div>
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
<div style="color:var(--text-muted);padding:1rem 0.25rem;">
|
||||
{% if filtered %}
|
||||
No lines match the current filter.
|
||||
{% else %}
|
||||
No logs collected yet for <code>{{ name }}</code>. Lines appear within a few
|
||||
seconds of the container writing to stdout/stderr — unless it's on the log
|
||||
exclude list or log collection is turned off in
|
||||
<a href="/settings/thresholds/">Settings → Thresholds & Retention</a>.
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
@@ -1,31 +0,0 @@
|
||||
{#- One line on a swarm service's task failures, from Swarm's own task records
|
||||
(agent ≥ 1.8.0): a crash loop or the failures of the last 24h, and how the
|
||||
newest troubled task ended, in Swarm's words. Expects `s` with state,
|
||||
crash_looping, failures_1h, failures_24h and last_error; `style` is
|
||||
optional extra CSS for the wrapper. Renders nothing for a healthy service
|
||||
with no recent failures. -#}
|
||||
{%- macro ago(sec) -%}
|
||||
{%- if sec < 90 -%}{{ sec }}s ago
|
||||
{%- elif sec < 5400 -%}{{ sec // 60 }}m ago
|
||||
{%- elif sec < 172800 -%}{{ sec // 3600 }}h ago
|
||||
{%- else -%}{{ sec // 86400 }}d ago{%- endif -%}
|
||||
{%- endmacro -%}
|
||||
{%- set e = s.last_error -%}
|
||||
{%- if s.crash_looping or s.failures_24h or (e and s.state != 'healthy') %}
|
||||
<div style="display:flex;flex-wrap:wrap;align-items:baseline;gap:0.2rem 0.6rem;font-size:0.78rem;color:var(--text-muted);{{ style or '' }}">
|
||||
{% if s.crash_looping %}
|
||||
<span style="color:var(--red);font-weight:600;">Crash-looping · {{ s.failures_1h }} tasks failed in the last hour</span>
|
||||
{% elif s.failures_24h %}
|
||||
<span style="color:var(--orange);">{{ s.failures_24h }} task{{ 's' if s.failures_24h != 1 }} failed in the last 24h</span>
|
||||
{% endif %}
|
||||
{% if e %}
|
||||
<span title="How Swarm recorded the end of this service's newest failed task">
|
||||
Last task {{ e.state }}:
|
||||
<span style="color:var(--text);font-family:ui-monospace,monospace;overflow-wrap:anywhere;">{{ e.error or e.message or e.state }}</span>
|
||||
{%- if e.exit_code %} · exit {{ e.exit_code }}{% endif %}
|
||||
{%- if e.node %} · {{ e.node }}{% endif %}
|
||||
{%- if e.age_s is not none %} · {{ ago(e.age_s) }}{% endif %}
|
||||
</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{%- endif %}
|
||||
@@ -26,7 +26,6 @@
|
||||
<div style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.5rem;">
|
||||
{{ container.status }}{% if uptime %} · up {{ uptime }}{% endif %}
|
||||
{% if host %} · on <a href="/hosts/{{ host.id }}" style="color:var(--text-muted);">{{ host.name }}</a>{% endif %}
|
||||
· <a href="/plugins/docker/container/{{ host_id }}/{{ name }}/logs">View logs</a>
|
||||
</div>
|
||||
|
||||
{# ── Facts grid ──────────────────────────────────────────────────────────── #}
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
{# docker/container_logs.html — per-container log viewer (m79) #}
|
||||
{% extends "base.html" %}
|
||||
{% from "_macros.html" import crumbs %}
|
||||
{% block title %}Logs — {{ name }} — Docker — Steward{% endblock %}
|
||||
{% block breadcrumb %}{{ crumbs([
|
||||
("Docker", "/plugins/docker/"),
|
||||
(name, "/plugins/docker/container/" ~ host_id ~ "/" ~ name),
|
||||
("Logs", "")]) }}{% endblock %}
|
||||
{% block content %}
|
||||
|
||||
<div style="display:flex;align-items:baseline;gap:0.6rem;margin-bottom:0.35rem;flex-wrap:wrap;">
|
||||
<h1 class="page-title" style="margin-bottom:0;">{{ name }}</h1>
|
||||
<span style="font-size:0.9rem;color:var(--text-muted);">logs</span>
|
||||
</div>
|
||||
<div style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1rem;">
|
||||
Recent lines (newest first) collected by the host agent{% if host %} on
|
||||
<a href="/hosts/{{ host.id }}" style="color:var(--text-muted);">{{ host.name }}</a>{% endif %} —
|
||||
updated every few seconds.
|
||||
<a href="/plugins/docker/container/{{ host_id }}/{{ name }}">← back to container</a>
|
||||
</div>
|
||||
|
||||
{# ── Controls: stream filter + text search (drive the fragment via HTMX) ───── #}
|
||||
<form id="log-controls" onsubmit="return false;"
|
||||
style="display:flex;gap:0.6rem;align-items:center;flex-wrap:wrap;margin-bottom:0.6rem;">
|
||||
<label style="font-size:0.8rem;color:var(--text-muted);display:flex;align-items:center;gap:0.35rem;">
|
||||
Stream
|
||||
<select name="stream" style="padding:0.25rem 0.4rem;">
|
||||
<option value="all">all</option>
|
||||
<option value="stdout">stdout</option>
|
||||
<option value="stderr">stderr</option>
|
||||
</select>
|
||||
</label>
|
||||
<input type="search" name="q" placeholder="Filter lines…" autocomplete="off"
|
||||
aria-label="Filter log lines"
|
||||
style="flex:1;min-width:180px;padding:0.3rem 0.5rem;">
|
||||
</form>
|
||||
|
||||
<div class="card-flush">
|
||||
<div id="log-lines"
|
||||
hx-get="/plugins/docker/container/{{ host_id }}/{{ name }}/logs/lines"
|
||||
hx-trigger="load, every 5s, change from:#log-controls, keyup changed delay:400ms from:#log-controls"
|
||||
hx-include="#log-controls"
|
||||
hx-swap="innerHTML"
|
||||
role="log" aria-live="polite" tabindex="0"
|
||||
style="max-height:70vh;overflow:auto;padding:0.5rem 0.75rem;
|
||||
font-family:ui-monospace,monospace;font-size:0.8rem;line-height:1.5;">
|
||||
<div style="color:var(--text-muted);">Loading…</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% endblock %}
|
||||
@@ -7,8 +7,8 @@
|
||||
|
||||
<h1 class="page-title" style="margin-bottom:0.4rem;">Image & disk usage</h1>
|
||||
<p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.5rem;">
|
||||
Reclaimable = space held by images no container references. Admins can prune
|
||||
per host below; each action runs an audited Ansible playbook on that host.
|
||||
Reclaimable = space held by images no container references. Cleanup actions
|
||||
(prune) arrive in a later release — these are read-only figures for now.
|
||||
</p>
|
||||
|
||||
{% if host_groups %}
|
||||
@@ -47,43 +47,6 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Cleanup actions (admin only; audited Ansible prune run) ───────────── #}
|
||||
{% if session.user_role == 'admin' %}
|
||||
<div style="margin-bottom:1rem;">
|
||||
{% if ansible_available and g.has_target %}
|
||||
<div style="display:flex;flex-wrap:wrap;gap:0.5rem;align-items:center;">
|
||||
<form method="post" action="/plugins/docker/disk/{{ g.host_id }}/prune" style="margin:0;"
|
||||
data-msg="Remove all STOPPED containers on {{ g.host_name|e }}? This cannot be undone."
|
||||
onsubmit="return confirm(this.dataset.msg);">
|
||||
<input type="hidden" name="target" value="containers">
|
||||
<button type="submit" class="btn btn-sm">Prune stopped containers</button>
|
||||
</form>
|
||||
<form method="post" action="/plugins/docker/disk/{{ g.host_id }}/prune" style="margin:0;"
|
||||
data-msg="Remove ALL unused images on {{ g.host_name|e }} (docker image prune -a)? Any image not used by a container is deleted."
|
||||
onsubmit="return confirm(this.dataset.msg);">
|
||||
<input type="hidden" name="target" value="images">
|
||||
<button type="submit" class="btn btn-sm">Prune unused images</button>
|
||||
</form>
|
||||
<form method="post" action="/plugins/docker/disk/{{ g.host_id }}/prune" style="margin:0;"
|
||||
data-msg="Run a full system prune on {{ g.host_name|e }}? Removes stopped containers, unused networks, dangling images and build cache."
|
||||
onsubmit="return confirm(this.dataset.msg);">
|
||||
<input type="hidden" name="target" value="system">
|
||||
<button type="submit" class="btn btn-sm btn-danger">System prune…</button>
|
||||
</form>
|
||||
</div>
|
||||
<div style="font-size:0.72rem;color:var(--text-muted);margin-top:0.4rem;">
|
||||
Reclaimed space appears on the next agent sample.
|
||||
</div>
|
||||
{% elif not ansible_available %}
|
||||
<div style="font-size:0.74rem;color:var(--text-muted);">Cleanup needs the Ansible runner (currently unavailable).</div>
|
||||
{% else %}
|
||||
<div style="font-size:0.74rem;color:var(--text-muted);">
|
||||
Link an <a href="/hosts/{{ g.host_id }}">Ansible target</a> to this host to enable prune actions.
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{# ── Per-image table ──────────────────────────────────────────────────── #}
|
||||
<div class="card-flush">
|
||||
<table class="table">
|
||||
|
||||
@@ -26,10 +26,9 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Swarm services: collapsed per-service, each current replica labelled with
|
||||
its host (tasks Swarm replaced are left out). Ghost replicas (tasks on nodes
|
||||
with no Steward agent) come from the managers' placement data so the picture
|
||||
is cluster-complete. A failing service names Swarm's last task error. ───── #}
|
||||
{# ── Swarm services: collapsed per-service, each replica labelled with its host.
|
||||
Ghost replicas (tasks on nodes with no Steward agent) come from the managers'
|
||||
placement data so the picture is cluster-complete. ──────────────────────── #}
|
||||
{% if swarm_services %}
|
||||
<div class="card-flush" style="margin-bottom:1.5rem;">
|
||||
<div style="display:flex;align-items:baseline;gap:0.6rem;padding:0.5rem 0.75rem;border-bottom:1px solid var(--border);">
|
||||
@@ -47,7 +46,6 @@
|
||||
{{ s.running }}/{{ s.desired }} running
|
||||
</span>
|
||||
</div>
|
||||
{% with style="padding-left:1.1rem;margin-bottom:0.35rem;" %}{% include "docker/_swarm_failures.html" %}{% endwith %}
|
||||
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(210px,1fr));gap:0.4rem;padding-left:1.1rem;">
|
||||
{% for r in s.replicas %}
|
||||
{% if r.ghost %}
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th><th>Task failures</th>
|
||||
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@@ -39,14 +39,9 @@
|
||||
<td style="font-size:0.78rem;color:var(--text-muted);">
|
||||
{% for p in s.placement %}{{ p.node }} ({{ p.running }}){% if not loop.last %}, {% endif %}{% else %}—{% endfor %}
|
||||
</td>
|
||||
<td style="max-width:360px;">
|
||||
{% include "docker/_swarm_failures.html" %}
|
||||
{%- if not s.tasks_known %}<span style="font-size:0.78rem;color:var(--text-dim);" title="The reporting manager's agent predates task reporting (1.8.0); update it to see task failures.">—</span>
|
||||
{%- elif not (s.crash_looping or s.failures_24h or (s.last_error and s.state != 'healthy')) %}<span style="font-size:0.78rem;color:var(--text-dim);">none in 24h</span>{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% else %}
|
||||
<tr><td colspan="6" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
|
||||
<tr><td colspan="5" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:var(--green);">{{ running_count }}</span>
|
||||
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">running</span>
|
||||
</div>
|
||||
<div title="Containers that died or were OOM-killed in the last 24h, plus swarm tasks that failed (a task Swarm replaced is not a failure)">
|
||||
<div title="Distinct containers with a die or OOM event in the last 24h">
|
||||
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:{% if failed_24h %}var(--red){% else %}var(--text-muted){% endif %};">{{ failed_24h }}</span>
|
||||
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">failed (24h)</span>
|
||||
</div>
|
||||
@@ -24,7 +24,6 @@
|
||||
<div style="display:flex;align-items:center;gap:0.4rem;font-size:0.82rem;">
|
||||
<span class="dot {% if s.state == 'healthy' %}dot-up{% elif s.state == 'degraded' %}dot-warn{% else %}dot-down{% endif %}"></span>
|
||||
<span style="font-weight:500;min-width:0;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;">{{ s.name }}</span>
|
||||
{% if s.crash_looping %}<span style="color:var(--red);font-size:0.7rem;font-weight:600;flex-shrink:0;" title="{{ s.failures_1h }} tasks failed in the last hour{% if s.last_error %}: {{ s.last_error.error or s.last_error.message or s.last_error.state }}{% endif %}">crash-looping</span>{% endif %}
|
||||
<span style="color:var(--text-muted);font-family:ui-monospace,monospace;font-size:0.74rem;margin-left:auto;flex-shrink:0;">{{ s.running }}/{{ s.desired }}</span>
|
||||
</div>
|
||||
<div style="display:flex;flex-wrap:wrap;gap:0.25rem;margin:0.2rem 0 0 1rem;">
|
||||
|
||||
@@ -21,13 +21,3 @@ def get_scheduled_tasks() -> list:
|
||||
def get_blueprint():
|
||||
from .routes import host_agent_bp
|
||||
return host_agent_bp
|
||||
|
||||
|
||||
def register_mcp_tools(register) -> None:
|
||||
from .mcp_tools import register_tools
|
||||
register_tools(register)
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
from .mcp_tools import mcp_overview as findings
|
||||
return await findings(db)
|
||||
|
||||
+11
-289
@@ -20,7 +20,7 @@ from collections import deque
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
|
||||
AGENT_VERSION = "1.8.0"
|
||||
AGENT_VERSION = "1.6.0"
|
||||
|
||||
# Default path to the local Docker Engine socket. Overridable via the
|
||||
# `docker_socket` config key; collection is silently skipped if it's absent or
|
||||
@@ -34,10 +34,7 @@ class ConfigError(Exception):
|
||||
|
||||
REQUIRED_KEYS = ("url", "token")
|
||||
INT_KEYS = ("interval_seconds",)
|
||||
LIST_KEYS = ("mounts", "docker_log_exclude")
|
||||
# Truthy strings for bool keys; anything else (incl. empty) is False.
|
||||
BOOL_KEYS = ("docker_logs_enabled",)
|
||||
_BOOL_TRUE = ("1", "true", "yes", "on")
|
||||
LIST_KEYS = ("mounts",)
|
||||
|
||||
|
||||
def read_config(path: str) -> dict:
|
||||
@@ -61,8 +58,6 @@ def read_config(path: str) -> dict:
|
||||
raise ConfigError(f"{path}:{lineno}: {key} must be int")
|
||||
elif key in LIST_KEYS:
|
||||
cfg[key] = [v.strip() for v in value.split(",") if v.strip()]
|
||||
elif key in BOOL_KEYS:
|
||||
cfg[key] = value.lower() in _BOOL_TRUE
|
||||
else:
|
||||
cfg[key] = value
|
||||
except FileNotFoundError:
|
||||
@@ -74,10 +69,6 @@ def read_config(path: str) -> dict:
|
||||
|
||||
cfg.setdefault("interval_seconds", 30)
|
||||
cfg.setdefault("docker_socket", DEFAULT_DOCKER_SOCKET)
|
||||
# Container-log collection is on by default (operator preference); a host can
|
||||
# opt out with `docker_logs_enabled = false` or thin it with docker_log_exclude.
|
||||
cfg.setdefault("docker_logs_enabled", True)
|
||||
cfg.setdefault("docker_log_exclude", [])
|
||||
return cfg
|
||||
|
||||
|
||||
@@ -412,15 +403,6 @@ def _rates(cur: dict, prev: dict, dt: float) -> dict:
|
||||
|
||||
DOCKER_API_TIMEOUT = 5.0
|
||||
|
||||
# Container-log collection (m79). Logs ride in the same push as metrics, so one
|
||||
# interval's batch is capped to keep a chatty container from bloating a POST (and
|
||||
# the backoff ring): once the cap is hit a marker line is emitted and collection
|
||||
# stops — the deferred lines come on the next interval. On a container's first
|
||||
# interval we seed from a short tail, then switch to an incremental since-cursor
|
||||
# kept per container in the agent's rate-state.
|
||||
DOCKER_LOG_BATCH_MAX_BYTES = 262144 # 256 KiB of log text per push, all containers
|
||||
DOCKER_LOG_FIRST_TAIL = 50 # lines seeded on a container's first interval
|
||||
|
||||
|
||||
def _dechunk(body: bytes) -> bytes:
|
||||
"""Decode an HTTP/1.1 chunked-transfer body into the raw payload."""
|
||||
@@ -668,198 +650,6 @@ def collect_docker(socket_path: str) -> list:
|
||||
return list(ex.map(lambda c: _collect_one_container(socket_path, c), containers))
|
||||
|
||||
|
||||
# ─── container logs (m79) ─────────────────────────────────────────────────────
|
||||
#
|
||||
# The logs endpoint returns a raw multiplexed byte stream, not JSON, so it needs
|
||||
# a sibling of _docker_request that hands back bytes. Non-TTY containers frame
|
||||
# stdout/stderr with an 8-byte header (stream byte + 4-byte big-endian length);
|
||||
# TTY containers emit unframed bytes. We request timestamps=1, so every line is
|
||||
# prefixed with an RFC3339Nano timestamp we parse for the since-cursor.
|
||||
|
||||
_LOG_STREAM_NAMES = {0: "stdout", 1: "stdout", 2: "stderr"}
|
||||
|
||||
|
||||
def _docker_request_raw(socket_path: str, path: str,
|
||||
timeout: float = DOCKER_API_TIMEOUT) -> bytes:
|
||||
"""GET `path` from the Docker API over the Unix socket; return the raw body.
|
||||
|
||||
Sibling of `_docker_request` for non-JSON endpoints (container logs). Reuses
|
||||
the connect / send / de-chunk scaffolding; raises OSError on any transport
|
||||
problem or non-2xx status so callers can silent-skip. A 200 with an empty
|
||||
body (no new lines since the cursor) returns b"".
|
||||
"""
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(timeout)
|
||||
try:
|
||||
sock.connect(socket_path)
|
||||
req = (
|
||||
"GET " + path + " HTTP/1.1\r\n"
|
||||
"Host: docker\r\n"
|
||||
"Accept: application/octet-stream\r\n"
|
||||
"Connection: close\r\n"
|
||||
"\r\n"
|
||||
)
|
||||
sock.sendall(req.encode("ascii"))
|
||||
chunks = []
|
||||
while True:
|
||||
buf = sock.recv(65536)
|
||||
if not buf:
|
||||
break
|
||||
chunks.append(buf)
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
raw = b"".join(chunks)
|
||||
head, _, body = raw.partition(b"\r\n\r\n")
|
||||
header_text = head.decode("latin-1")
|
||||
status_line = header_text.split("\r\n", 1)[0]
|
||||
parts = status_line.split(None, 2)
|
||||
status = int(parts[1]) if len(parts) >= 2 and parts[1].isdigit() else 0
|
||||
if not (200 <= status < 300):
|
||||
raise OSError(f"docker API {path} returned {status}")
|
||||
if "transfer-encoding: chunked" in header_text.lower():
|
||||
body = _dechunk(body)
|
||||
return body
|
||||
|
||||
|
||||
def _demux_docker_logs(raw: bytes) -> list:
|
||||
"""Split a Docker logs stream into [(stream_name, payload_bytes), …].
|
||||
|
||||
Non-TTY containers multiplex with an 8-byte frame header (stream byte in
|
||||
{0,1,2} + 4-byte big-endian length). The stream is treated as framed only if
|
||||
the whole buffer parses as clean back-to-back frames; anything else (a TTY
|
||||
container's raw stream, or a malformed header) falls back to a single stdout
|
||||
blob so no bytes are lost.
|
||||
"""
|
||||
if not raw:
|
||||
return []
|
||||
frames = []
|
||||
i, n = 0, len(raw)
|
||||
while i + 8 <= n:
|
||||
stream = raw[i]
|
||||
if stream > 2 or raw[i + 1] or raw[i + 2] or raw[i + 3]:
|
||||
break # not a valid frame header → not framed
|
||||
size = int.from_bytes(raw[i + 4:i + 8], "big")
|
||||
if i + 8 + size > n:
|
||||
break # frame overruns the buffer → not framed
|
||||
frames.append((_LOG_STREAM_NAMES.get(stream, "stdout"),
|
||||
raw[i + 8:i + 8 + size]))
|
||||
i += 8 + size
|
||||
if frames and i == n:
|
||||
return frames
|
||||
return [("stdout", raw)]
|
||||
|
||||
|
||||
def _parse_log_ts(token: str):
|
||||
"""Parse a Docker RFC3339Nano timestamp token into a datetime, or None.
|
||||
|
||||
Normalises `Z` → `+00:00` and trims Docker's nanosecond precision to the
|
||||
microseconds datetime.fromisoformat accepts (stdlib, Python 3.8+ safe).
|
||||
"""
|
||||
t = token.strip()
|
||||
if not t:
|
||||
return None
|
||||
if t.endswith("Z"):
|
||||
t = t[:-1] + "+00:00"
|
||||
m = re.match(r"^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(\.\d+)?(.*)$", t)
|
||||
if m:
|
||||
frac = m.group(2) or ""
|
||||
if len(frac) > 7: # "." + up to 6 fractional digits
|
||||
frac = frac[:7]
|
||||
t = m.group(1) + frac + m.group(3)
|
||||
try:
|
||||
return datetime.fromisoformat(t)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _parse_container_logs(raw: bytes) -> list:
|
||||
"""Demux + line-split a logs stream → [(dt|None, stream, line), …] in order.
|
||||
|
||||
Each line is timestamp-prefixed (we request timestamps=1); an unparseable
|
||||
prefix yields dt=None with the full raw line kept.
|
||||
"""
|
||||
out = []
|
||||
for stream, payload in _demux_docker_logs(raw):
|
||||
text = payload.decode("utf-8", "replace")
|
||||
for raw_line in text.split("\n"):
|
||||
if not raw_line:
|
||||
continue
|
||||
token, _, msg = raw_line.partition(" ")
|
||||
dt = _parse_log_ts(token)
|
||||
if dt is not None:
|
||||
out.append((dt, stream, msg))
|
||||
else:
|
||||
out.append((None, stream, raw_line))
|
||||
return out
|
||||
|
||||
|
||||
def collect_docker_logs(socket_path: str, containers: list, state: dict,
|
||||
exclude=None,
|
||||
max_bytes: int = DOCKER_LOG_BATCH_MAX_BYTES) -> list:
|
||||
"""New container-log lines since the last interval, for running containers.
|
||||
|
||||
Keeps a per-container since-cursor in `state["docker_log_cursors"]`
|
||||
(name → unix ts) so each interval fetches only what's new; a container's
|
||||
first interval seeds from a short tail. The whole batch is capped at
|
||||
`max_bytes` of log text — once exceeded a truncation marker is appended and
|
||||
collection stops (logs must never balloon a push). `exclude` container names
|
||||
are skipped entirely (a per-host bandwidth opt-out). Best-effort: a
|
||||
per-container transport error just contributes nothing this interval.
|
||||
Returns [{"container", "stream", "ts", "line"}, …].
|
||||
"""
|
||||
exclude_set = set(exclude or ())
|
||||
cursors = state.setdefault("docker_log_cursors", {})
|
||||
running = {c.get("name") for c in (containers or [])
|
||||
if c.get("name") and c.get("status") == "running"}
|
||||
# Forget cursors for containers no longer running so state can't grow without
|
||||
# bound over the agent's lifetime.
|
||||
for gone in [k for k in cursors if k not in running]:
|
||||
del cursors[gone]
|
||||
|
||||
out: list = []
|
||||
total = 0
|
||||
truncated = False
|
||||
for c in (containers or []):
|
||||
name = c.get("name")
|
||||
if not name or name not in running or name in exclude_set:
|
||||
continue
|
||||
prev = cursors.get(name)
|
||||
base = f"/containers/{name}/logs?stdout=1&stderr=1×tamps=1"
|
||||
# since accepts a fractional Unix ts; we still dedup the boundary line.
|
||||
path = (f"{base}&tail={DOCKER_LOG_FIRST_TAIL}" if prev is None
|
||||
else f"{base}&since={prev:.9f}")
|
||||
try:
|
||||
raw = _docker_request_raw(socket_path, path)
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
newest = prev
|
||||
for dt, stream, line in _parse_container_logs(raw):
|
||||
ts_unix = dt.timestamp() if dt is not None else None
|
||||
if prev is not None and ts_unix is not None and ts_unix <= prev:
|
||||
continue # boundary line already sent last interval
|
||||
if total >= max_bytes:
|
||||
truncated = True
|
||||
break
|
||||
out.append({"container": name, "stream": stream,
|
||||
"ts": dt.isoformat() if dt is not None else None,
|
||||
"line": line})
|
||||
total += len(line)
|
||||
if ts_unix is not None and (newest is None or ts_unix > newest):
|
||||
newest = ts_unix
|
||||
if newest is not None:
|
||||
cursors[name] = newest
|
||||
if truncated:
|
||||
break
|
||||
|
||||
if truncated:
|
||||
out.append({"container": "_steward", "stream": "stderr",
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"line": (f"[steward] log batch truncated at {max_bytes} bytes; "
|
||||
"remaining lines deferred to the next interval")})
|
||||
return out
|
||||
|
||||
|
||||
# ─── swarm (manager-only) ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -874,46 +664,14 @@ def _swarm_is_manager(info: dict) -> bool:
|
||||
return bool((info or {}).get("Swarm", {}).get("ControlAvailable"))
|
||||
|
||||
|
||||
# Newest tasks kept per service. Swarm retains about 5 per slot by default, so
|
||||
# this covers a replicated service's recent history and a crash loop's deaths
|
||||
# without shipping a global service's whole fleet every sample.
|
||||
SWARM_TASKS_PER_SERVICE = 10
|
||||
_TASK_TEXT_CHARS = 300
|
||||
|
||||
|
||||
def _swarm_task(t: dict) -> dict:
|
||||
"""One task as Steward stores it: where it ran, what Swarm wants of it,
|
||||
what it is doing, and why it stopped (Err/Message, exit code)."""
|
||||
status = t.get("Status") or {}
|
||||
cstatus = status.get("ContainerStatus") or {}
|
||||
exit_code = cstatus.get("ExitCode")
|
||||
return {
|
||||
"task_id": t.get("ID", "") or "",
|
||||
"slot": t.get("Slot"),
|
||||
"node_id": t.get("NodeID", "") or "",
|
||||
"desired_state": (t.get("DesiredState") or "").lower(),
|
||||
"state": (status.get("State") or "").lower(),
|
||||
"message": (status.get("Message") or "")[:_TASK_TEXT_CHARS],
|
||||
"error": (status.get("Err") or "")[:_TASK_TEXT_CHARS],
|
||||
"exit_code": exit_code if isinstance(exit_code, int) else None,
|
||||
"at": status.get("Timestamp") or t.get("UpdatedAt") or "",
|
||||
}
|
||||
|
||||
|
||||
def _swarm_services(services: list, tasks: list) -> list:
|
||||
"""Roll desired-vs-running replicas per service up from the task list.
|
||||
|
||||
Replica health isn't on the service object — it's derived by counting tasks:
|
||||
`running` = tasks currently in state running; `desired` = the service's
|
||||
configured replica count (replicated) or the count of tasks the scheduler
|
||||
wants running (global, which has no fixed number). Each service also
|
||||
carries its newest tasks (`tasks`), so Steward can tell a task Swarm
|
||||
replaced from one that crashed, and say why the last one failed.
|
||||
wants running (global, which has no fixed number).
|
||||
"""
|
||||
tasks_by_svc: dict = {}
|
||||
for t in (tasks or []):
|
||||
if t.get("ServiceID"):
|
||||
tasks_by_svc.setdefault(t["ServiceID"], []).append(_swarm_task(t))
|
||||
running_by_svc: dict = {}
|
||||
desired_by_svc: dict = {} # only used for global-mode services
|
||||
placement: dict = {} # svc_id -> {node_id: running_count}
|
||||
@@ -953,8 +711,6 @@ def _swarm_services(services: list, tasks: list) -> list:
|
||||
# sees every task, even ones whose containers live on other hosts).
|
||||
"placement": [{"node_id": n, "running": cnt}
|
||||
for n, cnt in sorted(placement.get(sid, {}).items())],
|
||||
"tasks": sorted(tasks_by_svc.get(sid, []), key=lambda t: t["at"],
|
||||
reverse=True)[:SWARM_TASKS_PER_SERVICE],
|
||||
})
|
||||
return out
|
||||
|
||||
@@ -1096,17 +852,13 @@ class RingBuffer:
|
||||
|
||||
|
||||
def build_sample(mounts: list[str], state: dict,
|
||||
docker_socket: str = DEFAULT_DOCKER_SOCKET, *,
|
||||
docker_logs_enabled: bool = True,
|
||||
docker_log_exclude=None) -> dict:
|
||||
docker_socket: str = DEFAULT_DOCKER_SOCKET) -> dict:
|
||||
"""Collect one full sample. Partial samples allowed if a collector fails.
|
||||
|
||||
`state` carries the previous network/disk counters + monotonic timestamp so
|
||||
throughput rates can be derived from deltas (and the per-container log
|
||||
since-cursors); it is mutated in place. `docker_socket` is probed
|
||||
best-effort — the `docker` key is omitted entirely when no containers are
|
||||
found, so non-Docker hosts add nothing to the payload. `docker_logs_enabled`
|
||||
/ `docker_log_exclude` gate incremental container-log collection.
|
||||
throughput rates can be derived from deltas; it is mutated in place.
|
||||
`docker_socket` is probed best-effort — the `docker` key is omitted entirely
|
||||
when no containers are found, so non-Docker hosts add nothing to the payload.
|
||||
"""
|
||||
sample: dict = {"ts": datetime.now(timezone.utc).isoformat()}
|
||||
try:
|
||||
@@ -1188,18 +940,6 @@ def build_sample(mounts: list[str], state: dict,
|
||||
if disk is not None:
|
||||
sample["docker_disk"] = disk
|
||||
|
||||
# Container logs (m79): incremental per-container tail folded into the same
|
||||
# push as metrics. Skipped when there are no containers or logging is off;
|
||||
# the key is omitted when nothing new arrived this interval.
|
||||
if docker and docker_logs_enabled:
|
||||
try:
|
||||
logs = collect_docker_logs(docker_socket, docker, state,
|
||||
exclude=docker_log_exclude)
|
||||
except Exception:
|
||||
logs = []
|
||||
if logs:
|
||||
sample["docker_logs"] = logs
|
||||
|
||||
return sample
|
||||
|
||||
|
||||
@@ -1218,18 +958,6 @@ def build_payload(samples: list[dict], hostname: str, metadata: dict) -> dict:
|
||||
BACKOFF_CAP = 300
|
||||
|
||||
|
||||
def _drop_logs(sample: dict) -> dict:
|
||||
"""Strip container logs from a sample before it's buffered for retry.
|
||||
|
||||
Logs are the one payload we never carry across a backoff — they'd bloat the
|
||||
ring buffer and go stale — while metrics are kept so an outage doesn't lose
|
||||
them. The since-cursor already advanced when the logs were collected, so the
|
||||
dropped lines are simply not re-sent (accepted loss during an outage).
|
||||
"""
|
||||
sample.pop("docker_logs", None)
|
||||
return sample
|
||||
|
||||
|
||||
def next_backoff(current: int) -> int:
|
||||
if current <= 0:
|
||||
return 30
|
||||
@@ -1294,8 +1022,6 @@ def main_loop(conf_path: str) -> int:
|
||||
hostname = cfg.get("hostname") or socket.gethostname()
|
||||
mounts = cfg.get("mounts") or default_mounts()
|
||||
docker_socket = cfg.get("docker_socket") or DEFAULT_DOCKER_SOCKET
|
||||
docker_logs_enabled = cfg.get("docker_logs_enabled", True)
|
||||
docker_log_exclude = cfg.get("docker_log_exclude") or []
|
||||
buffer = RingBuffer(maxlen=20)
|
||||
backoff = 0
|
||||
# Carries previous net/disk counters + monotonic ts for rate computation.
|
||||
@@ -1310,17 +1036,13 @@ def main_loop(conf_path: str) -> int:
|
||||
cfg = read_config(conf_path)
|
||||
mounts = cfg.get("mounts") or default_mounts()
|
||||
docker_socket = cfg.get("docker_socket") or DEFAULT_DOCKER_SOCKET
|
||||
docker_logs_enabled = cfg.get("docker_logs_enabled", True)
|
||||
docker_log_exclude = cfg.get("docker_log_exclude") or []
|
||||
metadata = collect_metadata() # refresh host_ip/distro on reload
|
||||
_log("INFO", "config reloaded")
|
||||
except ConfigError as e:
|
||||
_log("ERROR", f"reload failed: {e}")
|
||||
_reload_requested = False
|
||||
|
||||
sample = build_sample(mounts, rate_state, docker_socket,
|
||||
docker_logs_enabled=docker_logs_enabled,
|
||||
docker_log_exclude=docker_log_exclude)
|
||||
sample = build_sample(mounts, rate_state, docker_socket)
|
||||
buffered = buffer.drain()
|
||||
payload = build_payload(
|
||||
samples=buffered + [sample],
|
||||
@@ -1339,12 +1061,12 @@ def main_loop(conf_path: str) -> int:
|
||||
_log("ERROR", "server rejected payload (400) — dropping sample")
|
||||
elif status == 401:
|
||||
_log("ERROR", "token rejected (401) — check config + UI")
|
||||
buffer.push(_drop_logs(sample))
|
||||
buffer.push(sample)
|
||||
else:
|
||||
_log("WARN", f"POST failed (status={status}); buffering")
|
||||
for s in buffered:
|
||||
buffer.push(_drop_logs(s))
|
||||
buffer.push(_drop_logs(sample))
|
||||
buffer.push(s)
|
||||
buffer.push(sample)
|
||||
backoff = next_backoff(backoff)
|
||||
sleep_for = backoff
|
||||
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
"""The host_agent plugin's MCP tools and its get_overview findings."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.hosts import service as host_svc
|
||||
from steward.mcp._context import mcp_app
|
||||
from steward.mcp.tools._shape import age_s, db_session, iso, now_utc
|
||||
from steward.mcp.tools.overview import finding
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from .models import HostAgentRegistration
|
||||
|
||||
# A registration this old that has never checked in is a failed install, not one in progress.
|
||||
PENDING_GRACE = timedelta(minutes=30)
|
||||
|
||||
|
||||
def agent_state(reg, now, stale_after_s: int) -> str:
|
||||
"""The agent's state: reporting, stale (stopped checking in) or pending (never has)."""
|
||||
if reg.last_seen_at is None:
|
||||
return "pending"
|
||||
return "reporting" if (now - reg.last_seen_at).total_seconds() <= stale_after_s else "stale"
|
||||
|
||||
|
||||
async def _agents(db):
|
||||
return (await db.execute(
|
||||
select(HostAgentRegistration, Host)
|
||||
.join(Host, Host.id == HostAgentRegistration.host_id)
|
||||
.order_by(Host.name)
|
||||
)).all()
|
||||
|
||||
|
||||
async def list_agents() -> dict:
|
||||
"""Every host agent Steward has registered, with its version and whether it is reporting.
|
||||
|
||||
`state` is "reporting", "stale" (it checked in before but its last
|
||||
check-in is older than the stale window, `stale_after_s`) or "pending"
|
||||
(registered, never checked in, usually an install that is still running
|
||||
or that failed). Each row carries the agent version, kernel, distro, arch
|
||||
and the IP the agent reports. get_host shows an agent's latest metrics.
|
||||
"""
|
||||
now = now_utc()
|
||||
stale_after = host_svc.agent_stale_after_seconds(mcp_app().config)
|
||||
async with db_session() as db:
|
||||
rows = await _agents(db)
|
||||
agents = [
|
||||
{
|
||||
"host_id": h.id, "host_name": h.name,
|
||||
"state": agent_state(reg, now, stale_after),
|
||||
"last_seen_at": iso(reg.last_seen_at), "last_seen_age_s": age_s(reg.last_seen_at, now),
|
||||
"agent_version": reg.agent_version, "kernel": reg.kernel, "distro": reg.distro,
|
||||
"arch": reg.arch, "reported_ip": reg.host_ip,
|
||||
"registered_at": iso(reg.created_at),
|
||||
}
|
||||
for reg, h in rows
|
||||
]
|
||||
counts = {s: sum(1 for a in agents if a["state"] == s) for s in ("reporting", "stale", "pending")}
|
||||
return {"stale_after_s": stale_after, "counts": counts, "agents": agents, "total": len(agents)}
|
||||
|
||||
|
||||
async def mcp_overview(db) -> list[dict]:
|
||||
"""Agents registered more than PENDING_GRACE ago that have never checked in.
|
||||
|
||||
Agents that stopped reporting are already flagged by the core overview
|
||||
from their metrics, so they are not repeated here.
|
||||
"""
|
||||
now = now_utc()
|
||||
return [
|
||||
finding("warning", "agent_never_reported", h.name,
|
||||
"host agent was registered but has never checked in; the install may have failed",
|
||||
since=reg.created_at, ids={"host_id": h.id}, next="list_agents()")
|
||||
for reg, h in await _agents(db)
|
||||
if reg.last_seen_at is None and now - reg.created_at > PENDING_GRACE
|
||||
]
|
||||
|
||||
|
||||
def register_tools(register) -> None:
|
||||
register(list_agents, role=UserRole.viewer, read_only=True)
|
||||
@@ -202,7 +202,6 @@ async def ingest():
|
||||
accepted = 0
|
||||
latest_ts: datetime | None = None
|
||||
docker_snapshots: list[tuple[datetime, list]] = []
|
||||
docker_log_batches: list[tuple[datetime, list]] = []
|
||||
latest_swarm: dict | None = None
|
||||
latest_swarm_ts: datetime | None = None
|
||||
latest_disk: dict | None = None
|
||||
@@ -218,11 +217,6 @@ async def ingest():
|
||||
docker = sample.get("docker")
|
||||
if isinstance(docker, list) and docker:
|
||||
docker_snapshots.append((recorded_at, docker))
|
||||
# Container logs are time-series (append every line, not newest-only);
|
||||
# each record carries its own Docker ts, recorded_at is the fallback.
|
||||
docker_logs = sample.get("docker_logs")
|
||||
if isinstance(docker_logs, list) and docker_logs:
|
||||
docker_log_batches.append((recorded_at, docker_logs))
|
||||
# Swarm is current-state, not time-series — keep only the newest
|
||||
# sample's topology (a manager re-reports it every interval).
|
||||
swarm = sample.get("swarm")
|
||||
@@ -246,8 +240,7 @@ async def ingest():
|
||||
# (opportunistic synergy via the capability registry — no hard import,
|
||||
# no-op when docker is disabled). A failure here must never sink the
|
||||
# whole ingest, so the metrics above still land.
|
||||
if (docker_snapshots or latest_swarm is not None
|
||||
or latest_disk is not None or docker_log_batches):
|
||||
if docker_snapshots or latest_swarm is not None or latest_disk is not None:
|
||||
from steward.core.capabilities import has_capability, invoke_capability
|
||||
if has_capability("docker.persist_host_samples"):
|
||||
try:
|
||||
@@ -257,7 +250,7 @@ async def ingest():
|
||||
await invoke_capability(
|
||||
"docker.persist_host_samples", UserRole.admin,
|
||||
session, host, docker_snapshots, latest_swarm,
|
||||
latest_disk, docker_log_batches,
|
||||
latest_disk,
|
||||
)
|
||||
except Exception:
|
||||
current_app.logger.exception(
|
||||
|
||||
@@ -45,10 +45,6 @@ cat > "$CONF_FILE" <<EOF
|
||||
url = $STEWARD_URL
|
||||
token = $AGENT_TOKEN
|
||||
interval_seconds = 30
|
||||
# Container logs are collected by default. To opt this host out entirely:
|
||||
# docker_logs_enabled = false
|
||||
# To skip specific noisy containers (comma-separated names):
|
||||
# docker_log_exclude = watchtower, some-chatty-service
|
||||
EOF
|
||||
chown "root:$AGENT_USER" "$CONF_FILE"
|
||||
chmod 0640 "$CONF_FILE"
|
||||
|
||||
@@ -18,9 +18,6 @@ dependencies = [
|
||||
"click>=8.0",
|
||||
"httpx>=0.27",
|
||||
"cryptography>=42",
|
||||
# Floor, not a cap: MCPServer (mcp.server.mcpserver) and the mcp.types
|
||||
# mirror that steward/mcp/ is written against first ship in SDK 2.2.
|
||||
"mcp>=2.2",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
||||
@@ -1,108 +0,0 @@
|
||||
"""AI access: personal API tokens for MCP clients.
|
||||
|
||||
Every logged-in user manages their own tokens here, whatever their role — a
|
||||
token acts as its user, capped at the role it was given. These routes are
|
||||
session-only: a share-link viewer has no user to mint tokens for, and no token
|
||||
can reach them (MCP tools never mint or revoke tokens).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from quart import Blueprint, abort, current_app, redirect, render_template, request
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.auth.middleware import current_user_id, require_role
|
||||
from steward.core import api_tokens
|
||||
from steward.core.audit import log_audit
|
||||
from steward.core.settings import public_base_url
|
||||
from steward.mcp.server import MCP_PATH
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
ai_access_bp = Blueprint("ai_access", __name__, url_prefix="/ai-access")
|
||||
|
||||
_EXPIRY_CHOICES = [("", "Never"), ("30", "30 days"), ("90", "90 days"), ("365", "1 year")]
|
||||
|
||||
|
||||
def _mcp_url() -> str:
|
||||
"""The /mcp endpoint as a client outside this request reaches it."""
|
||||
return public_base_url(request) + MCP_PATH
|
||||
|
||||
|
||||
async def _session_user(db) -> User:
|
||||
user_id = current_user_id()
|
||||
if not user_id:
|
||||
abort(403)
|
||||
user = (await db.execute(select(User).where(User.id == user_id))).scalar_one_or_none()
|
||||
if user is None or not user.is_active:
|
||||
abort(403)
|
||||
return user
|
||||
|
||||
|
||||
async def _render_index(db, user: User, *, error: str | None = None, status: int = 200):
|
||||
tokens = await api_tokens.list_tokens(db, user.id)
|
||||
now = datetime.now(timezone.utc)
|
||||
rows = [{"token": t, "state": api_tokens.token_state(t, now)} for t in tokens]
|
||||
body = await render_template(
|
||||
"ai_access/index.html",
|
||||
rows=rows,
|
||||
user_role=user.role,
|
||||
grantable=api_tokens.grantable_roles(user.role),
|
||||
expiry_choices=_EXPIRY_CHOICES,
|
||||
error=error,
|
||||
mcp_url=_mcp_url(),
|
||||
)
|
||||
return body, status
|
||||
|
||||
|
||||
@ai_access_bp.get("/")
|
||||
@require_role(UserRole.viewer)
|
||||
async def index():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
user = await _session_user(db)
|
||||
return await _render_index(db, user)
|
||||
|
||||
|
||||
@ai_access_bp.post("/tokens")
|
||||
@require_role(UserRole.viewer)
|
||||
async def create():
|
||||
form = await request.form
|
||||
name = form.get("name", "")
|
||||
expires = form.get("expires_in_days", "")
|
||||
try:
|
||||
role = UserRole(form.get("role", UserRole.viewer.value))
|
||||
except ValueError:
|
||||
role = None
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
user = await _session_user(db)
|
||||
if role is None:
|
||||
return await _render_index(db, user, error="Pick a role for the token.", status=400)
|
||||
try:
|
||||
raw, token = await api_tokens.create_token(
|
||||
db, user_id=user.id, user_role=user.role, name=name, role=role,
|
||||
expires_in_days=int(expires) if expires.isdigit() and int(expires) > 0 else None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
return await _render_index(db, user, error=str(exc), status=400)
|
||||
await db.commit()
|
||||
|
||||
await log_audit(current_app, user.id, user.username, "api_token.create",
|
||||
entity_type="api_token", entity_id=str(token.id),
|
||||
detail={"name": token.name, "role": token.role.value})
|
||||
return await render_template("ai_access/created.html", raw_token=raw, token=token,
|
||||
mcp_url=_mcp_url())
|
||||
|
||||
|
||||
@ai_access_bp.post("/tokens/<int:token_id>/revoke")
|
||||
@require_role(UserRole.viewer)
|
||||
async def revoke(token_id: int):
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
user = await _session_user(db)
|
||||
token = await api_tokens.revoke_token(db, user_id=user.id, token_id=token_id)
|
||||
await db.commit()
|
||||
if token is None:
|
||||
abort(404)
|
||||
await log_audit(current_app, user.id, user.username, "api_token.revoke",
|
||||
entity_type="api_token", entity_id=str(token_id),
|
||||
detail={"name": token.name})
|
||||
return redirect("/ai-access/")
|
||||
+146
-60
@@ -4,34 +4,64 @@ from quart import Blueprint, render_template, request, redirect, url_for, curren
|
||||
from steward.core.audit import log_audit
|
||||
from sqlalchemy import select
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.ansible import sources as src_module
|
||||
from steward.models.alerts import AlertRule, AlertOperator, MaintenanceWindow
|
||||
from steward.models.alerts import AlertRule, AlertState, AlertStateEnum, AlertOperator, MaintenanceWindow
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.models.users import UserRole
|
||||
|
||||
alerts_bp = Blueprint("alerts", __name__, url_prefix="/alerts")
|
||||
|
||||
_OPERATORS = [(op.value, op.value) for op in AlertOperator]
|
||||
|
||||
def _rule_form(form) -> dict:
|
||||
"""The rule form's fields, as alert_svc.rule_fields takes them."""
|
||||
return {
|
||||
"name": form.get("name", ""),
|
||||
"source_module": form.get("source_module", ""),
|
||||
"resource_name": form.get("resource_name", ""),
|
||||
"metric_name": form.get("metric_name", ""),
|
||||
"operator": form.get("operator", ""),
|
||||
"threshold": form.get("threshold"),
|
||||
"consecutive_failures_required": form.get("consecutive_failures_required"),
|
||||
}
|
||||
# Static catalog: source_module → available metric names.
|
||||
# Unified monitors emit metrics under their TYPE (icmp/tcp/dns/http), each with
|
||||
# is_up (1.0 up / 0.0 down) and response_ms (where the probe times it).
|
||||
METRIC_CATALOG: dict[str, list[str]] = {
|
||||
"icmp": ["is_up", "response_ms"],
|
||||
"tcp": ["is_up", "response_ms"],
|
||||
"dns": ["is_up", "response_ms"],
|
||||
"http": ["is_up", "response_ms"],
|
||||
"traefik": ["request_rate", "error_rate", "latency_p50_ms", "latency_p95_ms",
|
||||
"latency_p99_ms", "response_bytes_rate", "cert_expiry_days"],
|
||||
"unifi": ["is_up", "latency_ms", "total_clients"],
|
||||
# restart_count = cumulative restarts (alert on crash-looping); is_healthy =
|
||||
# 1.0 healthy / 0.0 unhealthy from the container HEALTHCHECK (alert on <1).
|
||||
"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy"],
|
||||
"host_agent": [
|
||||
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
|
||||
"disk_used_pct_worst", "load_1m", "load_5m", "load_15m", "uptime_secs",
|
||||
],
|
||||
# snmp metrics are user-defined OID labels — populated dynamically from plugin_metrics
|
||||
"snmp": [],
|
||||
}
|
||||
|
||||
# Ordered list for source module picker
|
||||
_SOURCE_MODULES = list(METRIC_CATALOG.keys())
|
||||
|
||||
|
||||
def _form_time(value) -> datetime:
|
||||
"""A datetime-local field, read as UTC."""
|
||||
try:
|
||||
return datetime.fromisoformat((value or "").strip()).replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
raise ValueError(f"{value!r} is not a date and time.") from None
|
||||
async def _get_resources(db, source_module: str) -> list[str]:
|
||||
"""Return sorted distinct resource names for a source module.
|
||||
|
||||
For monitor types (icmp/tcp/dns/http), supplements with monitor names so
|
||||
the list is populated even before any metrics have been recorded.
|
||||
"""
|
||||
result = await db.execute(
|
||||
select(PluginMetric.resource_name)
|
||||
.where(PluginMetric.source_module == source_module)
|
||||
.distinct()
|
||||
.order_by(PluginMetric.resource_name)
|
||||
)
|
||||
resources: set[str] = {r for (r,) in result}
|
||||
|
||||
if source_module in ("icmp", "tcp", "dns", "http"):
|
||||
from steward.models.monitors import Monitor
|
||||
mon_result = await db.execute(
|
||||
select(Monitor.name).where(Monitor.type == source_module).order_by(Monitor.name)
|
||||
)
|
||||
for (name,) in mon_result:
|
||||
resources.add(name)
|
||||
|
||||
return sorted(resources)
|
||||
|
||||
|
||||
def _is_admin() -> bool:
|
||||
@@ -93,8 +123,20 @@ def _parse_ansible_action(form) -> tuple[dict | None, str | None]:
|
||||
@require_role(UserRole.viewer)
|
||||
async def list_alerts():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
active = await alert_svc.active_alerts(db)
|
||||
rules = await alert_svc.list_rules(db)
|
||||
result = await db.execute(
|
||||
select(AlertState, AlertRule)
|
||||
.join(AlertRule, AlertState.rule_id == AlertRule.id)
|
||||
.where(AlertState.state.in_([
|
||||
AlertStateEnum.firing, AlertStateEnum.acknowledged, AlertStateEnum.pending
|
||||
]))
|
||||
.order_by(AlertState.last_transition_at.desc())
|
||||
)
|
||||
active = result.all()
|
||||
|
||||
rules_result = await db.execute(
|
||||
select(AlertRule).order_by(AlertRule.name)
|
||||
)
|
||||
rules = rules_result.scalars().all()
|
||||
|
||||
return await render_template("alerts/list.html", active=active, rules=rules, operators=_OPERATORS)
|
||||
|
||||
@@ -106,9 +148,18 @@ async def rule_fields_api():
|
||||
current_resource = request.args.get("current_resource", "")
|
||||
current_metric = request.args.get("current_metric", "")
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
resources = await alert_svc.resources_for(db, source) if source else []
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
metrics = await alert_svc.metrics_for(db, source)
|
||||
resources = await _get_resources(db, source) if source else []
|
||||
metrics = METRIC_CATALOG.get(source, [])
|
||||
# Dynamic sources (e.g. snmp) have empty static catalog — query live metric names
|
||||
if source and not metrics:
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
m_result = await db.execute(
|
||||
select(PluginMetric.metric_name)
|
||||
.where(PluginMetric.source_module == source)
|
||||
.distinct()
|
||||
.order_by(PluginMetric.metric_name)
|
||||
)
|
||||
metrics = [r for (r,) in m_result]
|
||||
return await render_template(
|
||||
"alerts/_rule_fields.html",
|
||||
resources=resources,
|
||||
@@ -121,17 +172,17 @@ async def rule_fields_api():
|
||||
@alerts_bp.get("/rules/new")
|
||||
@require_role(UserRole.operator)
|
||||
async def new_rule():
|
||||
first_source = alert_svc.SOURCE_MODULES[0]
|
||||
first_source = _SOURCE_MODULES[0]
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
resources = await alert_svc.resources_for(db, first_source)
|
||||
resources = await _get_resources(db, first_source)
|
||||
return await render_template(
|
||||
"alerts/rules_form.html",
|
||||
rule=None,
|
||||
operators=_OPERATORS,
|
||||
source_modules=alert_svc.SOURCE_MODULES,
|
||||
source_modules=_SOURCE_MODULES,
|
||||
initial_source=first_source,
|
||||
resources=resources,
|
||||
metrics=alert_svc.METRIC_CATALOG.get(first_source, []),
|
||||
metrics=METRIC_CATALOG.get(first_source, []),
|
||||
ansible_sources=_ansible_source_names(),
|
||||
)
|
||||
|
||||
@@ -144,15 +195,15 @@ async def edit_rule(rule_id: str):
|
||||
rule = result.scalar_one_or_none()
|
||||
if rule is None:
|
||||
return "Not found", 404
|
||||
resources = await alert_svc.resources_for(db, rule.source_module)
|
||||
resources = await _get_resources(db, rule.source_module)
|
||||
return await render_template(
|
||||
"alerts/rules_form.html",
|
||||
rule=rule,
|
||||
operators=_OPERATORS,
|
||||
source_modules=alert_svc.SOURCE_MODULES,
|
||||
source_modules=_SOURCE_MODULES,
|
||||
initial_source=rule.source_module,
|
||||
resources=resources,
|
||||
metrics=alert_svc.METRIC_CATALOG.get(rule.source_module, []),
|
||||
metrics=METRIC_CATALOG.get(rule.source_module, []),
|
||||
ansible_sources=_ansible_source_names(),
|
||||
)
|
||||
|
||||
@@ -162,20 +213,36 @@ async def edit_rule(rule_id: str):
|
||||
async def create_rule():
|
||||
form = await request.form
|
||||
user_id = session.get("user_id")
|
||||
now = datetime.now(timezone.utc)
|
||||
# Ansible action is admin-only; non-admins simply can't set one.
|
||||
ansible_action = None
|
||||
if _is_admin():
|
||||
ansible_action, err = _parse_ansible_action(form)
|
||||
if err:
|
||||
return err, 400
|
||||
rule = AlertRule(
|
||||
name=form["name"].strip(),
|
||||
source_module=form["source_module"].strip(),
|
||||
resource_name=form["resource_name"].strip(),
|
||||
metric_name=form["metric_name"].strip(),
|
||||
operator=AlertOperator(form["operator"]),
|
||||
threshold=float(form["threshold"]),
|
||||
consecutive_failures_required=int(form.get("consecutive_failures_required") or 1),
|
||||
enabled=True,
|
||||
ansible_action=ansible_action,
|
||||
created_by=user_id,
|
||||
)
|
||||
state = AlertState(
|
||||
rule_id=rule.id,
|
||||
state=AlertStateEnum.inactive,
|
||||
last_transition_at=now,
|
||||
)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
try:
|
||||
fields = await alert_svc.rule_fields(db, **_rule_form(form))
|
||||
except ValueError as exc:
|
||||
return str(exc), 400
|
||||
rule = await alert_svc.add_rule(db, fields, created_by=user_id,
|
||||
ansible_action=ansible_action)
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
state.rule_id = rule.id
|
||||
db.add(state)
|
||||
await log_audit(current_app, user_id, session.get("username", ""), "alert_rule.created",
|
||||
entity_type="alert_rule", entity_id=rule.name,
|
||||
detail={"source": rule.source_module, "resource": rule.resource_name,
|
||||
@@ -200,12 +267,13 @@ async def update_rule(rule_id: str):
|
||||
rule = result.scalar_one_or_none()
|
||||
if rule is None:
|
||||
return "Not found", 404
|
||||
try:
|
||||
fields = await alert_svc.rule_fields(db, **_rule_form(form))
|
||||
except ValueError as exc:
|
||||
return str(exc), 400
|
||||
for k, v in fields.items():
|
||||
setattr(rule, k, v)
|
||||
rule.name = form["name"].strip()
|
||||
rule.source_module = form["source_module"].strip()
|
||||
rule.resource_name = form["resource_name"].strip()
|
||||
rule.metric_name = form["metric_name"].strip()
|
||||
rule.operator = AlertOperator(form["operator"])
|
||||
rule.threshold = float(form["threshold"])
|
||||
rule.consecutive_failures_required = int(form.get("consecutive_failures_required") or 1)
|
||||
if apply_action:
|
||||
rule.ansible_action = ansible_action
|
||||
await log_audit(current_app, session.get("user_id"), session.get("username", ""),
|
||||
@@ -251,12 +319,15 @@ async def delete_rule(rule_id: str):
|
||||
async def list_maintenance():
|
||||
now = datetime.now(timezone.utc)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
windows = await alert_svc.list_maintenance_windows(db)
|
||||
result = await db.execute(
|
||||
select(MaintenanceWindow).order_by(MaintenanceWindow.start_at.desc())
|
||||
)
|
||||
windows = result.scalars().all()
|
||||
return await render_template(
|
||||
"alerts/maintenance.html",
|
||||
windows=windows,
|
||||
now=now,
|
||||
source_modules=alert_svc.SOURCE_MODULES,
|
||||
source_modules=_SOURCE_MODULES,
|
||||
)
|
||||
|
||||
|
||||
@@ -266,7 +337,7 @@ async def new_maintenance():
|
||||
return await render_template(
|
||||
"alerts/maintenance_form.html",
|
||||
window=None,
|
||||
source_modules=alert_svc.SOURCE_MODULES,
|
||||
source_modules=_SOURCE_MODULES,
|
||||
)
|
||||
|
||||
|
||||
@@ -275,18 +346,26 @@ async def new_maintenance():
|
||||
async def create_maintenance():
|
||||
form = await request.form
|
||||
user_id = session.get("user_id")
|
||||
try:
|
||||
fields = alert_svc.window_fields(
|
||||
name=form.get("name", ""),
|
||||
start_at=_form_time(form.get("start_at")), end_at=_form_time(form.get("end_at")),
|
||||
scope_source=form.get("scope_source"), scope_resource=form.get("scope_resource"))
|
||||
except ValueError as exc:
|
||||
return str(exc), 400
|
||||
window = MaintenanceWindow(created_by=user_id, **fields)
|
||||
start_at = datetime.fromisoformat(form["start_at"]).replace(tzinfo=timezone.utc)
|
||||
end_at = datetime.fromisoformat(form["end_at"]).replace(tzinfo=timezone.utc)
|
||||
scope_source = form.get("scope_source", "").strip() or None
|
||||
scope_resource = form.get("scope_resource", "").strip() or None
|
||||
if scope_source is None:
|
||||
scope_resource = None # can't have resource without source
|
||||
window = MaintenanceWindow(
|
||||
name=form["name"].strip(),
|
||||
start_at=start_at,
|
||||
end_at=end_at,
|
||||
scope_source=scope_source,
|
||||
scope_resource=scope_resource,
|
||||
created_by=user_id,
|
||||
)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
db.add(window)
|
||||
scope_desc = alert_svc.window_scope(window)
|
||||
scope_desc = "all" if not scope_source else (
|
||||
f"{scope_source}/{scope_resource}" if scope_resource else f"{scope_source}/*"
|
||||
)
|
||||
await log_audit(current_app, user_id, session.get("username", ""),
|
||||
"maintenance_window.created", entity_type="maintenance_window",
|
||||
entity_id=window.name, detail={"scope": scope_desc})
|
||||
@@ -317,12 +396,19 @@ async def delete_maintenance(window_id: str):
|
||||
@require_role(UserRole.operator)
|
||||
async def acknowledge(rule_id: str):
|
||||
user_id = session.get("user_id")
|
||||
now = datetime.now(timezone.utc)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
try:
|
||||
await alert_svc.acknowledge(db, rule_id, user_id)
|
||||
except LookupError:
|
||||
result = await db.execute(
|
||||
select(AlertState).where(AlertState.rule_id == rule_id)
|
||||
)
|
||||
state = result.scalar_one_or_none()
|
||||
if state is None:
|
||||
return "Not found", 404
|
||||
except ValueError as exc:
|
||||
return f"Conflict: {exc}", 409
|
||||
if state.state != AlertStateEnum.firing:
|
||||
return "Conflict: alert is not in FIRING state", 409
|
||||
state.state = AlertStateEnum.acknowledged
|
||||
state.acknowledged_by = user_id
|
||||
state.acknowledged_at = now
|
||||
state.last_transition_at = now
|
||||
return redirect(url_for("alerts.list_alerts"))
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
"""Alert queries and validation shared by the alerts pages and the MCP tools.
|
||||
|
||||
Each function takes an open session (or plain values), with no Quart request
|
||||
context, so the routes and the /mcp tools (which run outside Quart's
|
||||
pipeline) call the same code and refuse the same input for the same reasons.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.models.alerts import (
|
||||
AlertOperator, AlertRule, AlertState, AlertStateEnum, MaintenanceWindow,
|
||||
)
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.models.monitors import MONITOR_TYPE_VALUES, Monitor
|
||||
|
||||
# Static catalog: source_module → available metric names.
|
||||
# Unified monitors emit metrics under their TYPE (icmp/tcp/dns/http), each with
|
||||
# is_up (1.0 up / 0.0 down) and response_ms (where the probe times it).
|
||||
METRIC_CATALOG: dict[str, list[str]] = {
|
||||
"icmp": ["is_up", "response_ms"],
|
||||
"tcp": ["is_up", "response_ms"],
|
||||
"dns": ["is_up", "response_ms"],
|
||||
"http": ["is_up", "response_ms"],
|
||||
"traefik": ["request_rate", "error_rate", "latency_p50_ms", "latency_p95_ms",
|
||||
"latency_p99_ms", "response_bytes_rate", "cert_expiry_days"],
|
||||
"unifi": ["is_up", "latency_ms", "total_clients"],
|
||||
# restart_count = cumulative restarts (alert on crash-looping); is_healthy =
|
||||
# 1.0 healthy / 0.0 unhealthy from the container HEALTHCHECK (alert on <1).
|
||||
"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy"],
|
||||
"host_agent": [
|
||||
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
|
||||
"disk_used_pct_worst", "load_1m", "load_5m", "load_15m", "uptime_secs",
|
||||
],
|
||||
# snmp metrics are user-defined OID labels — populated dynamically from plugin_metrics
|
||||
"snmp": [],
|
||||
}
|
||||
|
||||
# Ordered list for the source module picker.
|
||||
SOURCE_MODULES = list(METRIC_CATALOG.keys())
|
||||
|
||||
# States that need someone's attention: pending (breaching, not yet for long
|
||||
# enough), firing, and acknowledged-but-still-breaching.
|
||||
ACTIVE_STATES = (AlertStateEnum.firing, AlertStateEnum.acknowledged, AlertStateEnum.pending)
|
||||
|
||||
|
||||
async def active_alerts(db) -> list:
|
||||
"""(AlertState, AlertRule) rows not yet resolved, latest transition first."""
|
||||
return list((await db.execute(
|
||||
select(AlertState, AlertRule)
|
||||
.join(AlertRule, AlertState.rule_id == AlertRule.id)
|
||||
.where(AlertState.state.in_(ACTIVE_STATES))
|
||||
.order_by(AlertState.last_transition_at.desc())
|
||||
)).all())
|
||||
|
||||
|
||||
async def list_rules(db) -> list[AlertRule]:
|
||||
return list((await db.execute(select(AlertRule).order_by(AlertRule.name))).scalars())
|
||||
|
||||
|
||||
async def list_maintenance_windows(db) -> list[MaintenanceWindow]:
|
||||
"""Every window, latest start first."""
|
||||
return list((await db.execute(
|
||||
select(MaintenanceWindow).order_by(MaintenanceWindow.start_at.desc())
|
||||
)).scalars())
|
||||
|
||||
|
||||
async def resources_for(db, source_module: str) -> list[str]:
|
||||
"""Sorted distinct resource names for a source module.
|
||||
|
||||
For monitor types (icmp/tcp/dns/http), supplements with monitor names so
|
||||
the list is populated even before any metrics have been recorded.
|
||||
"""
|
||||
result = await db.execute(
|
||||
select(PluginMetric.resource_name)
|
||||
.where(PluginMetric.source_module == source_module)
|
||||
.distinct()
|
||||
.order_by(PluginMetric.resource_name)
|
||||
)
|
||||
resources: set[str] = {r for (r,) in result}
|
||||
if source_module in MONITOR_TYPE_VALUES:
|
||||
mon_result = await db.execute(
|
||||
select(Monitor.name).where(Monitor.type == source_module).order_by(Monitor.name))
|
||||
resources.update(name for (name,) in mon_result)
|
||||
return sorted(resources)
|
||||
|
||||
|
||||
async def metrics_for(db, source_module: str) -> list[str]:
|
||||
"""The metric names a rule on this source can watch: the static catalog, or
|
||||
for dynamic sources (snmp) the names recorded so far."""
|
||||
metrics = METRIC_CATALOG.get(source_module, [])
|
||||
if source_module and not metrics:
|
||||
result = await db.execute(
|
||||
select(PluginMetric.metric_name)
|
||||
.where(PluginMetric.source_module == source_module)
|
||||
.distinct()
|
||||
.order_by(PluginMetric.metric_name))
|
||||
metrics = [r for (r,) in result]
|
||||
return metrics
|
||||
|
||||
|
||||
def _choices(values: list[str], limit: int = 20) -> str:
|
||||
if not values:
|
||||
return "none yet"
|
||||
shown = ", ".join(values[:limit])
|
||||
return shown + (f" (and {len(values) - limit} more)" if len(values) > limit else "")
|
||||
|
||||
|
||||
async def rule_fields(db, *, name: str, source_module: str, resource_name: str, metric_name: str,
|
||||
operator: str, threshold, consecutive_failures_required=1) -> dict:
|
||||
"""Validated column values for a new or edited alert rule.
|
||||
|
||||
The same choices the rule form offers: a known source, a resource that
|
||||
source has reported (or, for monitors, a monitor of that type), and a
|
||||
metric it emits. ValueError says what was wrong and what is valid.
|
||||
"""
|
||||
name = (name or "").strip()
|
||||
if not name:
|
||||
raise ValueError("an alert rule needs a name.")
|
||||
source_module = (source_module or "").strip()
|
||||
if source_module not in METRIC_CATALOG:
|
||||
raise ValueError(f"source_module {source_module!r} is not one of {', '.join(SOURCE_MODULES)}.")
|
||||
resource_name = (resource_name or "").strip()
|
||||
resources = await resources_for(db, source_module)
|
||||
if resource_name not in resources:
|
||||
raise ValueError(f"{source_module} has no resource {resource_name!r}. "
|
||||
f"Known: {_choices(resources)}.")
|
||||
metric_name = (metric_name or "").strip()
|
||||
metrics = await metrics_for(db, source_module)
|
||||
if metric_name not in metrics:
|
||||
raise ValueError(f"{source_module} does not report {metric_name!r}. "
|
||||
f"Its metrics: {_choices(metrics)}.")
|
||||
try:
|
||||
op = AlertOperator((operator or "").strip())
|
||||
except ValueError:
|
||||
raise ValueError(f"operator {operator!r} is not one of "
|
||||
f"{', '.join(o.value for o in AlertOperator)}.") from None
|
||||
try:
|
||||
threshold = float(threshold)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"threshold must be a number, got {threshold!r}.") from None
|
||||
try:
|
||||
consecutive = (1 if consecutive_failures_required in (None, "")
|
||||
else int(consecutive_failures_required))
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("consecutive checks must be a whole number.") from None
|
||||
if consecutive < 1:
|
||||
raise ValueError("consecutive checks must be at least 1.")
|
||||
return {"name": name[:128], "source_module": source_module, "resource_name": resource_name,
|
||||
"metric_name": metric_name, "operator": op, "threshold": threshold,
|
||||
"consecutive_failures_required": consecutive}
|
||||
|
||||
|
||||
async def add_rule(db, fields: dict, *, created_by: str, enabled: bool = True,
|
||||
ansible_action: dict | None = None) -> AlertRule:
|
||||
"""Insert a rule and its inactive state row, in the caller's transaction."""
|
||||
rule = AlertRule(enabled=enabled, ansible_action=ansible_action, created_by=created_by, **fields)
|
||||
db.add(rule)
|
||||
await db.flush()
|
||||
db.add(AlertState(rule_id=rule.id, state=AlertStateEnum.inactive,
|
||||
last_transition_at=datetime.now(timezone.utc)))
|
||||
return rule
|
||||
|
||||
|
||||
async def acknowledge(db, rule_id: str, user_id: str) -> AlertState:
|
||||
"""Acknowledge a firing alert, in the caller's transaction.
|
||||
|
||||
LookupError when the rule has no state; ValueError when it is not firing.
|
||||
"""
|
||||
state = (await db.execute(
|
||||
select(AlertState).where(AlertState.rule_id == rule_id))).scalar_one_or_none()
|
||||
if state is None:
|
||||
raise LookupError(rule_id)
|
||||
if state.state != AlertStateEnum.firing:
|
||||
raise ValueError(f"the alert is {state.state.value}, not firing; only a firing alert "
|
||||
"can be acknowledged.")
|
||||
now = datetime.now(timezone.utc)
|
||||
state.state = AlertStateEnum.acknowledged
|
||||
state.acknowledged_by = user_id
|
||||
state.acknowledged_at = now
|
||||
state.last_transition_at = now
|
||||
return state
|
||||
|
||||
|
||||
def window_fields(*, name: str, start_at: datetime, end_at: datetime,
|
||||
scope_source: str | None, scope_resource: str | None) -> dict:
|
||||
"""Validated columns for a maintenance window. A resource needs a source."""
|
||||
name = (name or "").strip()
|
||||
if not name:
|
||||
raise ValueError("a maintenance window needs a name.")
|
||||
if end_at <= start_at:
|
||||
raise ValueError("the window must end after it starts.")
|
||||
scope_source = (scope_source or "").strip() or None
|
||||
scope_resource = (scope_resource or "").strip() or None
|
||||
if scope_source is not None and scope_source not in METRIC_CATALOG:
|
||||
raise ValueError(f"source {scope_source!r} is not one of {', '.join(SOURCE_MODULES)}.")
|
||||
if scope_source is None:
|
||||
scope_resource = None # can't have resource without source
|
||||
return {"name": name[:128], "start_at": start_at, "end_at": end_at,
|
||||
"scope_source": scope_source, "scope_resource": scope_resource}
|
||||
|
||||
|
||||
def window_scope(w) -> str:
|
||||
if not w.scope_source:
|
||||
return "all"
|
||||
return f"{w.scope_source}/{w.scope_resource}" if w.scope_resource else f"{w.scope_source}/*"
|
||||
@@ -1,64 +1,29 @@
|
||||
---
|
||||
# description: Reclaim disk on Docker / Swarm nodes — prune stopped containers, unused images, or a full system prune.
|
||||
# description: Reclaim disk on Docker / Swarm nodes by pruning unused images, containers, networks and build cache.
|
||||
# steward:category: maintenance
|
||||
# steward:confirm: true
|
||||
# Reclaim disk on Docker / Docker Swarm nodes by removing unused data.
|
||||
# `prune_target` selects the scope (default `system` preserves the original
|
||||
# behavior for existing manual/scheduled callers that don't set it):
|
||||
# prune_target=containers remove stopped containers only (docker container prune)
|
||||
# prune_target=images remove unused images (docker image prune;
|
||||
# + prune_all_images=true → -a, i.e. ALL unused, not just dangling)
|
||||
# prune_target=system docker system prune (dangling images, stopped
|
||||
# containers, unused networks, build cache). Widen with:
|
||||
# prune_all_images=true also ALL unused images
|
||||
# prune_volumes=true also unused named volumes (data loss risk)
|
||||
- name: Docker prune
|
||||
# Safe by default: prunes dangling images, stopped containers, unused networks
|
||||
# and build cache. Set extra-vars to widen scope:
|
||||
# prune_all_images=true also remove ALL unused images (not just dangling)
|
||||
# prune_volumes=true also remove unused named volumes (data loss risk)
|
||||
- name: Docker system prune
|
||||
hosts: all
|
||||
gather_facts: false
|
||||
become: true
|
||||
vars:
|
||||
prune_target: system
|
||||
prune_all_images: false
|
||||
prune_volumes: false
|
||||
tasks:
|
||||
- name: Validate prune_target
|
||||
ansible.builtin.assert:
|
||||
that: prune_target in ['containers', 'images', 'system']
|
||||
fail_msg: "prune_target must be one of: containers, images, system (got '{{ prune_target }}')"
|
||||
quiet: true
|
||||
|
||||
# ── Stopped containers only ───────────────────────────────────────────────
|
||||
- name: Prune stopped containers
|
||||
ansible.builtin.command:
|
||||
argv: ['docker', 'container', 'prune', '-f']
|
||||
register: container_prune
|
||||
changed_when: "'Total reclaimed space: 0B' not in container_prune.stdout"
|
||||
when: prune_target == 'containers'
|
||||
|
||||
# ── Unused images (dangling, or all unused with -a) ───────────────────────
|
||||
- name: Prune unused images
|
||||
ansible.builtin.command:
|
||||
argv: >-
|
||||
{{ ['docker', 'image', 'prune', '-f']
|
||||
+ (['-a'] if prune_all_images | bool else []) }}
|
||||
register: image_prune
|
||||
changed_when: "'Total reclaimed space: 0B' not in image_prune.stdout"
|
||||
when: prune_target == 'images'
|
||||
|
||||
# ── Full system prune (default) ───────────────────────────────────────────
|
||||
- name: Run docker system prune
|
||||
ansible.builtin.command:
|
||||
argv: >-
|
||||
{{ ['docker', 'system', 'prune', '-f']
|
||||
+ (['-a'] if prune_all_images | bool else [])
|
||||
+ (['--volumes'] if prune_volumes | bool else []) }}
|
||||
register: system_prune
|
||||
changed_when: "'Total reclaimed space: 0B' not in system_prune.stdout"
|
||||
when: prune_target == 'system'
|
||||
register: prune_result
|
||||
changed_when: "'Total reclaimed space: 0B' not in prune_result.stdout"
|
||||
|
||||
- name: Report reclaimed space
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ ((container_prune.stdout_lines | default([]))
|
||||
+ (image_prune.stdout_lines | default([]))
|
||||
+ (system_prune.stdout_lines | default([]))) | select | list }}
|
||||
msg: "{{ prune_result.stdout_lines | select | list }}"
|
||||
|
||||
@@ -106,39 +106,3 @@ async def fetch_scope_targets(db, scope: str) -> list:
|
||||
|
||||
result = await db.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
def describe_scope(scope: str, targets: dict[str, str], groups: dict[str, str]) -> str:
|
||||
"""A run's inventory scope in words: "target web-1", "group Docker Swarm",
|
||||
"all inventory targets", "<source>/<inventory file>". `targets` and
|
||||
`groups` map id → name; a scope naming a deleted one says so."""
|
||||
scope = scope or ""
|
||||
if scope == "steward:all":
|
||||
return "all inventory targets"
|
||||
for prefix, kind, names in (("steward:target:", "target", targets),
|
||||
("steward:group:", "group", groups)):
|
||||
if scope.startswith(prefix):
|
||||
ident = scope[len(prefix):]
|
||||
name = names.get(ident)
|
||||
return f"{kind} {name}" if name else f"{kind} {ident[:8]} (deleted)"
|
||||
if scope.startswith("repo:"):
|
||||
source, _, inv = scope[len("repo:"):].rpartition(":")
|
||||
return f"{source}/{inv}" if source else inv
|
||||
return scope
|
||||
|
||||
|
||||
async def scope_labels(db, scopes) -> dict[str, str]:
|
||||
"""{scope: describe_scope(scope)} for the given scopes, in two queries."""
|
||||
from sqlalchemy import select
|
||||
from steward.models.ansible_inventory import AnsibleGroup, AnsibleTarget
|
||||
|
||||
scopes = {s for s in scopes if s}
|
||||
t_ids = {s[len("steward:target:"):] for s in scopes if s.startswith("steward:target:")}
|
||||
g_ids = {s[len("steward:group:"):] for s in scopes if s.startswith("steward:group:")}
|
||||
targets = dict((await db.execute(
|
||||
select(AnsibleTarget.id, AnsibleTarget.name).where(AnsibleTarget.id.in_(t_ids))
|
||||
)).all()) if t_ids else {}
|
||||
groups = dict((await db.execute(
|
||||
select(AnsibleGroup.id, AnsibleGroup.name).where(AnsibleGroup.id.in_(g_ids))
|
||||
)).all()) if g_ids else {}
|
||||
return {s: describe_scope(s, targets, groups) for s in scopes}
|
||||
|
||||
@@ -9,7 +9,6 @@ from quart import (
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.ansible import executor, sources as src_module
|
||||
from steward.ansible.inventory_gen import scope_labels
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.ansible_schedule import AnsibleSchedule
|
||||
@@ -307,10 +306,8 @@ async def run_detail(run_id: str):
|
||||
res = await db.execute(
|
||||
select(User.username).where(User.id == run.triggered_by))
|
||||
triggered_label = res.scalar_one_or_none() or "(deleted user)"
|
||||
scope_label = (await scope_labels(db, [run.inventory_scope])).get(run.inventory_scope)
|
||||
return await render_template(
|
||||
"ansible/run_detail.html", run=run, triggered_label=triggered_label,
|
||||
scope_label=scope_label)
|
||||
"ansible/run_detail.html", run=run, triggered_label=triggered_label)
|
||||
|
||||
|
||||
@ansible_bp.post("/runs/<run_id>/cancel")
|
||||
@@ -371,7 +368,6 @@ async def schedules():
|
||||
r.id: r for r in (await db.execute(
|
||||
select(AnsibleRun).where(AnsibleRun.id.in_(run_ids)))).scalars().all()
|
||||
} if run_ids else {}
|
||||
labels = await scope_labels(db, [s.inventory_scope for s in scheds])
|
||||
|
||||
source_data = [
|
||||
{"name": s["name"], "playbooks": src_module.discover_playbooks(s["path"])}
|
||||
@@ -384,7 +380,6 @@ async def schedules():
|
||||
rows.append({
|
||||
"s": s,
|
||||
"last_run": runs.get(s.last_run_id) if s.last_run_id else None,
|
||||
"scope_label": labels.get(s.inventory_scope, s.inventory_scope),
|
||||
"next_run": (s.last_run_at + timedelta(seconds=s.interval_seconds)) if s.last_run_at else None,
|
||||
})
|
||||
if editing_id and s.id == editing_id:
|
||||
|
||||
+2
-50
@@ -3,12 +3,9 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from quart import Quart, render_template, request
|
||||
from quart import Quart, render_template
|
||||
from .config import load_bootstrap
|
||||
from .database import (
|
||||
init_db, ensure_database_reachable, DatabaseUnavailable,
|
||||
DB_CONNECT_TIMEOUT_SECONDS,
|
||||
)
|
||||
from .database import init_db
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -26,7 +23,6 @@ def create_app(
|
||||
bootstrap = {
|
||||
"database_url": "postgresql+asyncpg://test/test",
|
||||
"secret_key": "test-secret-key",
|
||||
"db_connect_timeout": DB_CONNECT_TIMEOUT_SECONDS,
|
||||
"plugin_dirs": ["plugins"],
|
||||
"plugin_install_dir": "plugins",
|
||||
}
|
||||
@@ -46,19 +42,6 @@ def create_app(
|
||||
from unittest.mock import MagicMock
|
||||
app.db_sessionmaker = MagicMock()
|
||||
|
||||
# ── 2b. Block until the database is actually reachable ────────────────────
|
||||
# Everything from here down touches the DB synchronously (migrations,
|
||||
# secret re-encryption, settings load). Gate all of it behind one readiness
|
||||
# check so a DB that is merely slow to come up — WAL recovery after an
|
||||
# unclean shutdown, or Docker DNS not yet serving the `db` name after a host
|
||||
# reboot — is waited out instead of crashing the container on a traceback.
|
||||
if not testing:
|
||||
from .database import wait_for_database
|
||||
wait_for_database(
|
||||
app.config["DATABASE_URL"],
|
||||
timeout_seconds=bootstrap["db_connect_timeout"],
|
||||
)
|
||||
|
||||
# ── 3. Core migrations only (creates app_settings table) ──────────────────
|
||||
if not testing:
|
||||
from .core.migration_runner import run_core_migrations
|
||||
@@ -150,7 +133,6 @@ def create_app(
|
||||
from .ansible.inventory_routes import inventory_bp
|
||||
from .settings.routes import settings_bp
|
||||
from .audit.routes import audit_bp
|
||||
from .ai_access.routes import ai_access_bp
|
||||
|
||||
app.register_blueprint(auth_bp)
|
||||
app.register_blueprint(dashboard_bp)
|
||||
@@ -162,7 +144,6 @@ def create_app(
|
||||
app.register_blueprint(inventory_bp)
|
||||
app.register_blueprint(settings_bp)
|
||||
app.register_blueprint(audit_bp)
|
||||
app.register_blueprint(ai_access_bp)
|
||||
|
||||
# Register the unified Monitor status source for the Status page. Plugins
|
||||
# may register additional sources from setup(). Idempotent.
|
||||
@@ -250,29 +231,6 @@ def create_app(
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
# ── 11b. Database availability gate ────────────────────────────────────────
|
||||
# Every page in Steward reads the database, so a database that has gone away
|
||||
# under a running app turns each request into an opaque 500 (this is how a
|
||||
# login attempt surfaced a bare gaierror). Acquire a connection up front,
|
||||
# retrying briefly to ride out a restart, and answer honestly if it stays
|
||||
# down rather than failing deep inside a handler with a generic error.
|
||||
if not testing:
|
||||
@app.before_request
|
||||
async def _database_gate():
|
||||
# /health is a liveness probe for the container itself — it must
|
||||
# stay answerable while the database is down, or a restart loop
|
||||
# gets triggered by a dependency outage. Static files need no DB.
|
||||
if request.endpoint in ("health", "static"):
|
||||
return None
|
||||
try:
|
||||
await ensure_database_reachable(app._db_engine)
|
||||
except DatabaseUnavailable as exc:
|
||||
logger.error("Database unreachable while serving %s: %s",
|
||||
request.path, exc)
|
||||
return await render_template(
|
||||
"errors/database_unavailable.html"), 503
|
||||
return None
|
||||
|
||||
# ── 12. Error handlers ─────────────────────────────────────────────────────
|
||||
@app.errorhandler(404)
|
||||
async def not_found(_):
|
||||
@@ -282,12 +240,6 @@ def create_app(
|
||||
async def server_error(_):
|
||||
return await render_template("errors/500.html"), 500
|
||||
|
||||
# ── MCP endpoint (/mcp) ───────────────────────────────────────────────────
|
||||
# Mounted after plugins load so their register_mcp_tools() hooks are seen.
|
||||
# It wraps app.asgi_app: /mcp never enters Quart's pipeline (see mcp/server.py).
|
||||
from .mcp import mount_mcp
|
||||
mount_mcp(app)
|
||||
|
||||
# ── 11. Startup hook ───────────────────────────────────────────────────────
|
||||
@app.before_serving
|
||||
async def startup():
|
||||
|
||||
@@ -49,8 +49,6 @@ def load_bootstrap(config_path: Path | str | None = None) -> dict[str, Any]:
|
||||
|
||||
secret_key = _resolve_secret_key(raw)
|
||||
|
||||
db_connect_timeout = _resolve_db_connect_timeout(raw)
|
||||
|
||||
# Plugin discovery spans two roots (see load_plugins / migration_runner):
|
||||
# • bundled — first-party plugins shipped inside the image at repo-root
|
||||
# `plugins/`; they version atomically with core and are read-only at runtime.
|
||||
@@ -70,47 +68,12 @@ def load_bootstrap(config_path: Path | str | None = None) -> dict[str, Any]:
|
||||
return {
|
||||
"database_url": database_url,
|
||||
"secret_key": secret_key,
|
||||
"db_connect_timeout": db_connect_timeout,
|
||||
"plugin_dirs": plugin_dirs,
|
||||
# Installs/downloads target the external (writable, persistent) dir.
|
||||
"plugin_install_dir": external_plugin_dir or bundled_plugin_dir,
|
||||
}
|
||||
|
||||
|
||||
def _resolve_db_connect_timeout(raw: dict) -> float:
|
||||
"""How long to wait for the database at startup, in seconds.
|
||||
|
||||
Bootstrap-only by necessity: this governs reaching the DB, so it cannot
|
||||
itself be read from the DB like the rest of Steward's settings.
|
||||
|
||||
A deployment whose database is slower to come up than the default (a large
|
||||
cluster replaying WAL, a remote DB behind a link that takes a while) can
|
||||
raise it rather than crash-looping the container.
|
||||
"""
|
||||
from .database import DB_CONNECT_TIMEOUT_SECONDS
|
||||
|
||||
value = _env("DB_CONNECT_TIMEOUT") or raw.get("database", {}).get("connect_timeout")
|
||||
if value is None or value == "":
|
||||
return DB_CONNECT_TIMEOUT_SECONDS
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"Invalid database connect timeout %r — using default %.0fs",
|
||||
value, DB_CONNECT_TIMEOUT_SECONDS,
|
||||
)
|
||||
return DB_CONNECT_TIMEOUT_SECONDS
|
||||
if parsed <= 0:
|
||||
# 0/negative would mean "never wait", which is the broken behaviour this
|
||||
# setting exists to fix — treat it as a mistake, not as an opt-out.
|
||||
logger.warning(
|
||||
"Database connect timeout %.0fs is not positive — using default %.0fs",
|
||||
parsed, DB_CONNECT_TIMEOUT_SECONDS,
|
||||
)
|
||||
return DB_CONNECT_TIMEOUT_SECONDS
|
||||
return parsed
|
||||
|
||||
|
||||
def _resolve_secret_key(raw: dict) -> str:
|
||||
"""Resolve secret_key: env var → file → auto-generate.
|
||||
|
||||
|
||||
@@ -1,160 +0,0 @@
|
||||
"""steward/core/api_tokens.py
|
||||
|
||||
Personal API tokens: the credential an AI client (MCP) presents as
|
||||
`Authorization: Bearer stw_…` to act as its user.
|
||||
|
||||
Tokens are high-entropy (`secrets.token_urlsafe(32)`), so they are stored as a
|
||||
plain sha256 — bcrypt is for low-entropy passwords — the same scheme as
|
||||
dashboard share tokens and the host-agent ingest token. The plaintext exists
|
||||
only in the create response.
|
||||
|
||||
A token's `role` is a CEILING. Every resolution re-reads the user's current
|
||||
role and acts with the lower of the two, so demoting or disabling a user takes
|
||||
effect on their tokens at the next request with nothing to revoke by hand.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import hashlib
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from steward.auth.middleware import role_meets
|
||||
from steward.models.api_tokens import ApiToken
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
TOKEN_PREFIX = "stw_"
|
||||
# How many plaintext characters the UI keeps to tell tokens apart.
|
||||
_DISPLAY_CHARS = 12
|
||||
# last_used_at is a "roughly when" signal; don't write a row per MCP call.
|
||||
_LAST_USED_RESOLUTION = timedelta(seconds=60)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Caller:
|
||||
"""Who an authenticated token request acts as, with its effective role."""
|
||||
user_id: str
|
||||
username: str
|
||||
role: UserRole
|
||||
token_id: int
|
||||
|
||||
|
||||
def generate_token() -> str:
|
||||
return TOKEN_PREFIX + secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def hash_token(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def display_prefix(raw: str) -> str:
|
||||
return raw[:_DISPLAY_CHARS]
|
||||
|
||||
|
||||
def effective_role(ceiling: UserRole, user_role: UserRole) -> UserRole:
|
||||
"""The lower of a token's ceiling and its user's current role."""
|
||||
return ceiling if role_meets(user_role, ceiling) else user_role
|
||||
|
||||
|
||||
def grantable_roles(user_role: UserRole) -> list[UserRole]:
|
||||
"""Roles a user may give a new token: their own and anything below it."""
|
||||
return [r for r in (UserRole.viewer, UserRole.operator, UserRole.admin)
|
||||
if role_meets(user_role, r)]
|
||||
|
||||
|
||||
def token_state(token: ApiToken, now: datetime | None = None) -> str:
|
||||
"""'active', 'expired' or 'revoked' — for display and resolution alike."""
|
||||
now = now or datetime.now(timezone.utc)
|
||||
if token.revoked_at is not None:
|
||||
return "revoked"
|
||||
if token.expires_at is not None and token.expires_at <= now:
|
||||
return "expired"
|
||||
return "active"
|
||||
|
||||
|
||||
async def create_token(
|
||||
db, *, user_id: str, user_role: UserRole, name: str, role: UserRole,
|
||||
expires_in_days: int | None = None,
|
||||
) -> tuple[str, ApiToken]:
|
||||
"""Mint a token. Returns (plaintext, row); the plaintext is never stored.
|
||||
|
||||
Raises ValueError when the requested ceiling is above the user's own role
|
||||
or the name is empty. The caller owns the transaction and commits it.
|
||||
"""
|
||||
name = (name or "").strip()[:128]
|
||||
if not name:
|
||||
raise ValueError("Give the token a name, so you can tell it apart later.")
|
||||
if not role_meets(user_role, role):
|
||||
raise ValueError(
|
||||
f"A token can't carry a higher role than yours ({user_role.value})."
|
||||
)
|
||||
now = datetime.now(timezone.utc)
|
||||
raw = generate_token()
|
||||
token = ApiToken(
|
||||
user_id=user_id,
|
||||
name=name,
|
||||
token_hash=hash_token(raw),
|
||||
prefix=display_prefix(raw),
|
||||
role=role,
|
||||
created_at=now,
|
||||
expires_at=now + timedelta(days=expires_in_days) if expires_in_days else None,
|
||||
)
|
||||
db.add(token)
|
||||
await db.flush()
|
||||
return raw, token
|
||||
|
||||
|
||||
async def list_tokens(db, user_id: str) -> list[ApiToken]:
|
||||
result = await db.execute(
|
||||
select(ApiToken).where(ApiToken.user_id == user_id)
|
||||
.order_by(ApiToken.created_at.desc())
|
||||
)
|
||||
return list(result.scalars())
|
||||
|
||||
|
||||
async def revoke_token(db, *, user_id: str, token_id: int) -> ApiToken | None:
|
||||
"""Revoke one of the user's own tokens. Returns it, or None if not theirs."""
|
||||
result = await db.execute(
|
||||
select(ApiToken).where(ApiToken.id == token_id, ApiToken.user_id == user_id)
|
||||
)
|
||||
token = result.scalar_one_or_none()
|
||||
if token is not None and token.revoked_at is None:
|
||||
token.revoked_at = datetime.now(timezone.utc)
|
||||
return token
|
||||
|
||||
|
||||
async def resolve_token(app, raw: str | None) -> Caller | None:
|
||||
"""Resolve a bearer token's plaintext to the Caller it acts as.
|
||||
|
||||
None for a missing, unknown, revoked or expired token, or one whose user is
|
||||
disabled. Never raises on a bad token; a database failure propagates so
|
||||
the transport can report the outage rather than a misleading 401.
|
||||
"""
|
||||
if not raw or not raw.startswith(TOKEN_PREFIX):
|
||||
return None
|
||||
now = datetime.now(timezone.utc)
|
||||
async with app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
result = await db.execute(
|
||||
select(ApiToken, User)
|
||||
.join(User, User.id == ApiToken.user_id)
|
||||
.where(ApiToken.token_hash == hash_token(raw))
|
||||
)
|
||||
row = result.first()
|
||||
if row is None:
|
||||
return None
|
||||
token, user = row
|
||||
if token_state(token, now) != "active" or not user.is_active:
|
||||
return None
|
||||
if token.last_used_at is None or now - token.last_used_at >= _LAST_USED_RESOLUTION:
|
||||
await db.execute(
|
||||
update(ApiToken).where(ApiToken.id == token.id)
|
||||
.values(last_used_at=now)
|
||||
)
|
||||
return Caller(
|
||||
user_id=user.id,
|
||||
username=user.username,
|
||||
role=effective_role(token.role, user.role),
|
||||
token_id=token.id,
|
||||
)
|
||||
@@ -76,14 +76,10 @@ async def _run_docker_retention(session, now: datetime) -> None:
|
||||
raw_days = int(await get_setting(session, "docker.retention.metrics_raw_days") or 7)
|
||||
rollup_days = int(await get_setting(session, "docker.retention.metrics_rollup_days") or 90)
|
||||
events_days = int(await get_setting(session, "docker.retention.events_days") or 30)
|
||||
logs_days = int(await get_setting(session, "docker.logs.retention_days") or 3)
|
||||
logs_cap = int(
|
||||
await get_setting(session, "docker.logs.max_bytes_per_container") or 5_000_000)
|
||||
counts = await invoke_capability(
|
||||
"docker.run_retention", UserRole.viewer, session,
|
||||
events_days=events_days, metrics_raw_days=raw_days,
|
||||
metrics_rollup_days=rollup_days, logs_retention_days=logs_days,
|
||||
logs_max_bytes_per_container=logs_cap, now=now,
|
||||
metrics_rollup_days=rollup_days, now=now,
|
||||
)
|
||||
if counts and any(counts.values()):
|
||||
logger.info("Docker retention: %s", counts)
|
||||
|
||||
@@ -1,122 +0,0 @@
|
||||
"""Reads over the generic metric store (plugin_metrics + its hourly rollup).
|
||||
|
||||
Any source that records into PluginMetric — host_agent, traefik, unifi, snmp,
|
||||
the monitor types, Steward's self-monitor — is readable through these without
|
||||
importing the plugin that wrote it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import func, or_, select, text
|
||||
|
||||
from steward.core.time_range import bucket_seconds
|
||||
from steward.models.metrics import PluginMetric, PluginMetricHourly
|
||||
|
||||
# Raw rows older than this are rolled into plugin_metrics_hourly and deleted
|
||||
# (see core/metrics_retention.py), so a range reaching past it reads the rollup.
|
||||
RAW_DAYS = 7
|
||||
|
||||
# Loose index scans: step through the distinct leading values of the
|
||||
# (source_module, resource_name, …) index one probe at a time instead of
|
||||
# scanning every sample. Postgres has no native skip scan before 18.
|
||||
_SOURCES_SQL = text("""
|
||||
WITH RECURSIVE s(v) AS (
|
||||
SELECT min(source_module) FROM plugin_metrics
|
||||
UNION ALL
|
||||
SELECT (SELECT min(source_module) FROM plugin_metrics WHERE source_module > s.v)
|
||||
FROM s WHERE s.v IS NOT NULL
|
||||
)
|
||||
SELECT v FROM s WHERE v IS NOT NULL
|
||||
""")
|
||||
|
||||
_RESOURCES_SQL = text("""
|
||||
WITH RECURSIVE r(v) AS (
|
||||
SELECT min(resource_name) FROM plugin_metrics WHERE source_module = :src
|
||||
UNION ALL
|
||||
SELECT (SELECT min(resource_name) FROM plugin_metrics
|
||||
WHERE source_module = :src AND resource_name > r.v)
|
||||
FROM r WHERE r.v IS NOT NULL
|
||||
)
|
||||
SELECT v FROM r WHERE v IS NOT NULL
|
||||
""")
|
||||
|
||||
|
||||
async def sources(db) -> list[str]:
|
||||
"""Every source_module with raw samples on record."""
|
||||
return [v for (v,) in await db.execute(_SOURCES_SQL)]
|
||||
|
||||
|
||||
async def resources(db, source_module: str) -> list[str]:
|
||||
"""Every resource a source has raw samples for, sorted."""
|
||||
return [v for (v,) in await db.execute(_RESOURCES_SQL, {"src": source_module})]
|
||||
|
||||
|
||||
async def latest_values(db, source_module: str, resource: str, *,
|
||||
include_children: bool = False) -> list[PluginMetric]:
|
||||
"""The newest sample of each metric for a resource, sorted by (resource, metric).
|
||||
|
||||
include_children also returns the resource's sub-resources, named
|
||||
"<resource>:<part>" (a host's mounts, interfaces, sensors).
|
||||
"""
|
||||
match = PluginMetric.resource_name == resource
|
||||
if include_children:
|
||||
like = resource.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + ":%"
|
||||
match = or_(match, PluginMetric.resource_name.like(like, escape="\\"))
|
||||
return list((await db.execute(
|
||||
select(PluginMetric)
|
||||
.where(PluginMetric.source_module == source_module, match)
|
||||
.distinct(PluginMetric.resource_name, PluginMetric.metric_name)
|
||||
.order_by(PluginMetric.resource_name, PluginMetric.metric_name,
|
||||
PluginMetric.recorded_at.desc())
|
||||
)).scalars())
|
||||
|
||||
|
||||
async def points(db, source_module: str, resource: str, metric: str, since: datetime,
|
||||
*, target: int = 120) -> list[tuple[datetime, float]]:
|
||||
"""(time, average) points for one metric since `since`, oldest first.
|
||||
|
||||
The part of the range older than the raw window comes from the hourly
|
||||
rollup; the rest is bucket-averaged in SQL (buckets no wider than an hour
|
||||
when both are present, so the series is evenly grained across the seam).
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
raw_cutoff = (now - timedelta(days=RAW_DAYS)).replace(minute=0, second=0, microsecond=0)
|
||||
out: list[tuple[datetime, float]] = []
|
||||
|
||||
if since < raw_cutoff:
|
||||
hourly = await db.execute(
|
||||
select(PluginMetricHourly.bucket, PluginMetricHourly.value_avg)
|
||||
.where(
|
||||
PluginMetricHourly.source_module == source_module,
|
||||
PluginMetricHourly.resource_name == resource,
|
||||
PluginMetricHourly.metric_name == metric,
|
||||
PluginMetricHourly.bucket >= since,
|
||||
PluginMetricHourly.bucket < raw_cutoff,
|
||||
)
|
||||
.order_by(PluginMetricHourly.bucket)
|
||||
)
|
||||
out.extend((b, float(v)) for b, v in hourly)
|
||||
|
||||
raw_since = max(since, raw_cutoff)
|
||||
width_s = bucket_seconds(raw_since, target)
|
||||
if since < raw_cutoff:
|
||||
width_s = min(width_s, 3600)
|
||||
bucket = func.date_bin(
|
||||
func.make_interval(0, 0, 0, 0, 0, 0, width_s),
|
||||
PluginMetric.recorded_at,
|
||||
func.to_timestamp(0),
|
||||
).label("bucket")
|
||||
raw = await db.execute(
|
||||
select(bucket, func.avg(PluginMetric.value))
|
||||
.where(
|
||||
PluginMetric.source_module == source_module,
|
||||
PluginMetric.resource_name == resource,
|
||||
PluginMetric.metric_name == metric,
|
||||
PluginMetric.recorded_at >= raw_since,
|
||||
)
|
||||
.group_by(bucket)
|
||||
.order_by(bucket)
|
||||
)
|
||||
out.extend((b, float(v)) for b, v in raw)
|
||||
return out
|
||||
@@ -29,11 +29,6 @@ _FAILED_PLUGINS: dict[str, str] = {}
|
||||
# {"plugin": <name>, "label": str, "href": str, "section": str}
|
||||
_PLUGIN_NAV: list[dict] = []
|
||||
|
||||
# Loaded plugin modules, by name, for the MCP server: a plugin contributes
|
||||
# agent tools through an optional register_mcp_tools(register) export, which
|
||||
# steward/mcp/server.py calls when it mounts (and on hot-reload, below).
|
||||
_PLUGIN_MODULES: dict[str, object] = {}
|
||||
|
||||
|
||||
def get_plugin_failures() -> dict[str, str]:
|
||||
"""Return a copy of the failed-plugin registry (name → error message)."""
|
||||
@@ -75,23 +70,6 @@ def _collect_plugin_nav(name: str, module) -> None:
|
||||
})
|
||||
|
||||
|
||||
def get_loaded_plugin_modules() -> dict[str, object]:
|
||||
"""Loaded plugin modules by name (a copy), for hooks consumed after loading."""
|
||||
return dict(_PLUGIN_MODULES)
|
||||
|
||||
|
||||
def _collect_plugin_module(app: "Quart", name: str, module) -> None:
|
||||
"""Remember a loaded plugin module; hand it to a live MCP server if mounted.
|
||||
|
||||
At boot the MCP server mounts after plugins load and reads them via
|
||||
get_loaded_plugin_modules(); a plugin hot-loaded later is registered here.
|
||||
"""
|
||||
_PLUGIN_MODULES[name] = module
|
||||
mcp_server = getattr(app, "mcp_server", None)
|
||||
if mcp_server is not None:
|
||||
mcp_server.register_plugin(name, module)
|
||||
|
||||
|
||||
def resolve_plugin_path(plugin_dirs: list[Path], name: str) -> Path | None:
|
||||
"""Return the first plugin root that contains `name`, else None.
|
||||
|
||||
@@ -109,8 +87,7 @@ def _import_plugin(name: str, plugin_path: Path):
|
||||
"""Load a plugin module by file path, avoiding sys.modules stdlib collisions.
|
||||
|
||||
Using importlib.import_module(name) fails for plugins whose names shadow
|
||||
Python stdlib modules (e.g. a plugin named 'json' or 'http' vs the stdlib
|
||||
package of the same name).
|
||||
Python stdlib modules (e.g. the 'http' plugin vs stdlib's 'http' package).
|
||||
This helper loads from the filesystem path directly and registers the module
|
||||
under a namespaced key so relative imports within the plugin still work.
|
||||
"""
|
||||
@@ -286,7 +263,6 @@ def load_plugins(app: "Quart") -> None:
|
||||
_FAILED_PLUGINS.pop(name, None) # clear any stale failure from a previous reload attempt
|
||||
_LOADED_PLUGINS.add(name)
|
||||
_collect_plugin_nav(name, module)
|
||||
_collect_plugin_module(app, name, module)
|
||||
logger.info("Plugin %r loaded (v%s)", name, meta.get("version", "?"))
|
||||
|
||||
|
||||
@@ -506,7 +482,6 @@ def hot_reload_plugin(app: "Quart", name: str) -> tuple[bool, str]:
|
||||
|
||||
_LOADED_PLUGINS.add(name)
|
||||
_collect_plugin_nav(name, module)
|
||||
_collect_plugin_module(app, name, module)
|
||||
logger.info("Plugin %r hot-reloaded (v%s)", name, meta.get("version", "?"))
|
||||
return True, f"Plugin activated (v{meta.get('version', '?')})"
|
||||
|
||||
|
||||
+10
-4
@@ -11,8 +11,7 @@ from email.message import EmailMessage
|
||||
|
||||
from sqlalchemy import and_, case, func, select
|
||||
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.models.alerts import AlertEvent
|
||||
from steward.models.alerts import AlertRule, AlertState, AlertStateEnum, AlertEvent
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.models.monitors import Monitor, MonitorResult
|
||||
|
||||
@@ -52,7 +51,14 @@ async def build_report_data(app) -> dict:
|
||||
]
|
||||
|
||||
# ── Active alerts ─────────────────────────────────────────────────────
|
||||
active_rows = await alert_svc.active_alerts(db)
|
||||
active_result = await db.execute(
|
||||
select(AlertState, AlertRule)
|
||||
.join(AlertRule, AlertState.rule_id == AlertRule.id)
|
||||
.where(AlertState.state.in_([
|
||||
AlertStateEnum.firing, AlertStateEnum.acknowledged, AlertStateEnum.pending,
|
||||
]))
|
||||
.order_by(AlertState.last_transition_at.desc())
|
||||
)
|
||||
active_alerts = [
|
||||
{
|
||||
"state": state.state.value.upper(),
|
||||
@@ -62,7 +68,7 @@ async def build_report_data(app) -> dict:
|
||||
"operator": rule.operator.value,
|
||||
"threshold": rule.threshold,
|
||||
}
|
||||
for state, rule in active_rows
|
||||
for state, rule in active_result.all()
|
||||
]
|
||||
|
||||
# ── Alert event counts (7d) ───────────────────────────────────────────
|
||||
|
||||
@@ -18,7 +18,6 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Iterable, Mapping
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
@@ -79,24 +78,12 @@ DEFAULTS: dict[str, Any] = {
|
||||
"thresholds.load_warn": 80, "thresholds.load_crit": 100,
|
||||
"thresholds.temp_warn": 70, "thresholds.temp_crit": 85,
|
||||
"thresholds.uptime_warn": 99.0, "thresholds.uptime_crit": 95.0,
|
||||
# Days left on an HTTPS monitor's certificate (lower is worse), and how
|
||||
# many container deaths within an hour count as crash-looping.
|
||||
"thresholds.tls_warn_days": 14, "thresholds.tls_crit_days": 3,
|
||||
"thresholds.crash_loop_deaths": 3,
|
||||
# Docker time-series retention (rule 25 — tunable, no restart). Raw 30s
|
||||
# samples are heavy, so keep a short raw window then roll up to hourly
|
||||
# averages kept much longer; lifecycle events are light, keep a month.
|
||||
"docker.retention.metrics_raw_days": 7,
|
||||
"docker.retention.metrics_rollup_days": 90,
|
||||
"docker.retention.events_days": 30,
|
||||
# Container logs (m79): on by default for every container (operator
|
||||
# preference). `exclude` names containers the server drops on ingest; the
|
||||
# per-container ring bounds storage (rotate oldest past whichever of ~age or
|
||||
# ~bytes hits first — a chatty container just keeps a shorter window).
|
||||
"docker.logs.enabled": True,
|
||||
"docker.logs.exclude": [],
|
||||
"docker.logs.retention_days": 3,
|
||||
"docker.logs.max_bytes_per_container": 5_000_000,
|
||||
# Host/plugin metrics retention (plugin_metrics): keep a short raw window at
|
||||
# the agent's ~30s cadence, then roll up to hourly averages kept much longer.
|
||||
"metrics.retention.raw_days": 7,
|
||||
@@ -111,6 +98,7 @@ DEFAULTS: dict[str, Any] = {
|
||||
# Per-plugin yaml config defaults are merged on top at load time.
|
||||
"plugin.docker": {"enabled": True},
|
||||
"plugin.host_agent": {"enabled": True},
|
||||
"plugin.http": {"enabled": True},
|
||||
"plugin.snmp": {"enabled": True},
|
||||
# OIDC single-sign-on
|
||||
"oidc.enabled": False,
|
||||
@@ -233,82 +221,6 @@ async def set_setting(session: AsyncSession, key: str, value: Any) -> None:
|
||||
_undecryptable_secrets.discard(key)
|
||||
|
||||
|
||||
async def delete_setting(session: AsyncSession, key: str) -> bool:
|
||||
"""Remove a stored setting row. Returns True if a row was actually deleted.
|
||||
|
||||
Call in a transaction. Deleting a key that also has a DEFAULT reverts it to
|
||||
that default rather than unsetting it — so this only meaningfully *removes*
|
||||
a setting when no default declares it.
|
||||
"""
|
||||
result = await session.execute(
|
||||
select(AppSetting).where(AppSetting.key == key)
|
||||
)
|
||||
row = result.scalar_one_or_none()
|
||||
if row is None:
|
||||
return False
|
||||
await session.delete(row)
|
||||
return True
|
||||
|
||||
|
||||
async def get_stored_plugin_names(session: AsyncSession) -> list[str]:
|
||||
"""Plugin names that have a real stored row, ignoring DEFAULTS.
|
||||
|
||||
Deliberately NOT derived from get_all_settings: that merges DEFAULTS in, and
|
||||
a plugin.* key present only as a default is a code-level declaration rather
|
||||
than operator data. Offering to "remove" such a key would be a lie — the
|
||||
default would simply reassert it on the next load (exactly the bug behind
|
||||
the phantom `plugin.http` entry). Only stored rows can actually be cleaned up.
|
||||
|
||||
Returns names only, never values: plugin config can hold credentials, and
|
||||
nothing that lists orphans needs to read them.
|
||||
"""
|
||||
prefix = "plugin."
|
||||
result = await session.execute(
|
||||
select(AppSetting.key).where(AppSetting.key.like(f"{prefix}%"))
|
||||
)
|
||||
return sorted(
|
||||
key[len(prefix):] for key in result.scalars() if key != prefix
|
||||
)
|
||||
|
||||
|
||||
def find_orphaned_plugins(
|
||||
stored_names: Iterable[str],
|
||||
installed_names: Iterable[str],
|
||||
failures: Mapping[str, str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Plugins that are configured or failed to load, but are not installed.
|
||||
|
||||
Pure so it can be tested without a DB or a filesystem. "Not installed" is
|
||||
intentionally not treated as "safe to delete" — an external plugin under
|
||||
/data/plugins can be missing merely because the volume isn't mounted or an
|
||||
install failed, so the caller surfaces these for an explicit operator
|
||||
decision instead of removing them automatically.
|
||||
|
||||
Driven by the UNION of stored settings and load failures, not by stored
|
||||
settings alone. `load_plugins` records a failure for any enabled plugin
|
||||
whose directory it cannot find, and the admin banner counts those — so a
|
||||
failure with no stored row (a plugin enabled by a DEFAULTS entry) would
|
||||
otherwise be counted in the banner and shown nowhere, which is exactly the
|
||||
dead end this section exists to close.
|
||||
|
||||
Each row carries `removable`: only a plugin with a real stored row can
|
||||
actually be cleaned up. Offering Remove for a default-declared plugin would
|
||||
be a lie — deleting nothing, while the default reasserts it on next load.
|
||||
"""
|
||||
installed = set(installed_names)
|
||||
failures = dict(failures or {})
|
||||
stored = {name for name in stored_names if name}
|
||||
candidates = (stored | set(failures)) - installed
|
||||
return [
|
||||
{
|
||||
"name": name,
|
||||
"reason": failures.get(name),
|
||||
"removable": name in stored,
|
||||
}
|
||||
for name in sorted(candidates)
|
||||
]
|
||||
|
||||
|
||||
async def get_all_settings(session: AsyncSession) -> dict[str, Any]:
|
||||
"""Return flat key→value dict with defaults filled in for missing keys."""
|
||||
result = await session.execute(select(AppSetting))
|
||||
@@ -383,8 +295,6 @@ def to_thresholds_cfg(settings: dict[str, Any]) -> dict:
|
||||
"temp": {"warn": g("thresholds.temp_warn", 70), "crit": g("thresholds.temp_crit", 85), "dir": "high"},
|
||||
"latency": {"warn": g("ping.threshold.good_ms", 50), "crit": g("ping.threshold.warn_ms", 200), "dir": "high"},
|
||||
"uptime": {"warn": g("thresholds.uptime_warn", 99.0), "crit": g("thresholds.uptime_crit", 95.0), "dir": "low"},
|
||||
"tls_days": {"warn": g("thresholds.tls_warn_days", 14), "crit": g("thresholds.tls_crit_days", 3), "dir": "low"},
|
||||
"crash_loop": {"warn": g("thresholds.crash_loop_deaths", 3), "crit": g("thresholds.crash_loop_deaths", 3), "dir": "high"},
|
||||
}
|
||||
|
||||
|
||||
|
||||
+2
-195
@@ -1,49 +1,10 @@
|
||||
from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
create_async_engine, async_sessionmaker, AsyncEngine, AsyncSession,
|
||||
)
|
||||
from sqlalchemy.pool import NullPool
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from quart import Quart
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Recycle pooled connections after 30 minutes. Nothing in Postgres closes an
|
||||
# idle connection by default, but the path to it is not always durable: NAT and
|
||||
# conntrack tables drop idle flows, and a Docker network rebuild silently
|
||||
# strands existing sockets. Capping connection age means a stranded connection
|
||||
# is retired on a timer instead of surfacing as a failed query later.
|
||||
POOL_RECYCLE_SECONDS = 1800
|
||||
|
||||
# Total budget for the database to become reachable at startup, and the backoff
|
||||
# ceiling between attempts. 60s comfortably covers Postgres WAL recovery after
|
||||
# an unclean shutdown (observed ~7s) plus Docker DNS coming up on a host reboot.
|
||||
DB_CONNECT_TIMEOUT_SECONDS = 60.0
|
||||
_BACKOFF_INITIAL_SECONDS = 0.5
|
||||
_BACKOFF_MAX_SECONDS = 5.0
|
||||
|
||||
# Per-REQUEST retry schedule (seconds between attempts), total ~1.75s across 4
|
||||
# tries. Far shorter than the startup budget on purpose: nobody is watching a
|
||||
# container boot, but somebody is watching this page load. Long enough to ride
|
||||
# out a pool blip or a fast reconnect, short enough that a genuinely-down
|
||||
# database gets an honest answer instead of a spinner.
|
||||
_REQUEST_RETRY_DELAYS = (0.25, 0.5, 1.0)
|
||||
|
||||
# Postgres SQLSTATEs that will never resolve by waiting — retrying these just
|
||||
# delays a clear error behind a full timeout budget.
|
||||
_FATAL_SQLSTATES = {
|
||||
"28P01", # invalid_password
|
||||
"28000", # invalid_authorization_specification
|
||||
"3D000", # invalid_catalog_name — database does not exist
|
||||
}
|
||||
|
||||
|
||||
def init_db(app: "Quart") -> None:
|
||||
"""Create async engine and attach db_sessionmaker to app.
|
||||
@@ -52,162 +13,8 @@ def init_db(app: "Quart") -> None:
|
||||
Does not create tables — Alembic handles migrations.
|
||||
"""
|
||||
db_url: str = app.config["DATABASE_URL"]
|
||||
engine = create_async_engine(
|
||||
db_url,
|
||||
echo=False,
|
||||
# Check a pooled connection is still alive before handing it out. When
|
||||
# the database restarts, every connection already in the pool is dead
|
||||
# at the socket level; without this, SQLAlchemy only discovers that by
|
||||
# failing a real query, so the first operation after a DB restart
|
||||
# errors out on whatever triggered it (a request, a scheduled poll).
|
||||
# The check is a cheap round-trip and it makes a DB restart invisible.
|
||||
pool_pre_ping=True,
|
||||
pool_recycle=POOL_RECYCLE_SECONDS,
|
||||
)
|
||||
engine = create_async_engine(db_url, echo=False)
|
||||
app.db_sessionmaker: async_sessionmaker[AsyncSession] = async_sessionmaker(
|
||||
engine, expire_on_commit=False
|
||||
)
|
||||
app._db_engine = engine
|
||||
|
||||
|
||||
def _fatal_sqlstate(exc: BaseException) -> str | None:
|
||||
"""Return the SQLSTATE if this error chain carries a non-retryable one."""
|
||||
seen: set[int] = set()
|
||||
cur: BaseException | None = exc
|
||||
while cur is not None and id(cur) not in seen:
|
||||
seen.add(id(cur))
|
||||
sqlstate = getattr(cur, "sqlstate", None)
|
||||
if sqlstate in _FATAL_SQLSTATES:
|
||||
return sqlstate
|
||||
cur = cur.__cause__ or cur.__context__
|
||||
return None
|
||||
|
||||
|
||||
def _describe(exc: BaseException) -> str:
|
||||
"""Innermost cause, which is the part that says what actually went wrong.
|
||||
|
||||
SQLAlchemy wraps DBAPI errors several layers deep; the outer message is
|
||||
boilerplate, so surface the root for the waiting-for-database log line.
|
||||
"""
|
||||
cur: BaseException = exc
|
||||
seen: set[int] = {id(cur)}
|
||||
while True:
|
||||
nxt = cur.__cause__ or cur.__context__
|
||||
if nxt is None or id(nxt) in seen:
|
||||
return f"{type(cur).__name__}: {cur}"
|
||||
seen.add(id(nxt))
|
||||
cur = nxt
|
||||
|
||||
|
||||
def wait_for_database(
|
||||
db_url: str,
|
||||
timeout_seconds: float = DB_CONNECT_TIMEOUT_SECONDS,
|
||||
) -> None:
|
||||
"""Block until the database accepts a connection, or raise after the budget.
|
||||
|
||||
Steward otherwise assumes the database is both resolvable and *ready* the
|
||||
first time it asks, which is false in two ordinary situations:
|
||||
|
||||
• Host reboot / full stack restart — the app container can try to resolve
|
||||
the `db` service name before Docker's embedded DNS has the record,
|
||||
giving `gaierror -2 Name or service not known`.
|
||||
• Unclean shutdown — Postgres is listening but still replaying WAL, and
|
||||
refuses connections with "the database system is not yet accepting
|
||||
connections" until recovery reaches a consistent state.
|
||||
|
||||
compose's `depends_on: service_healthy` covers ordering on a clean `up`,
|
||||
but not either of the above. Both are transient and self-healing, so retry
|
||||
rather than dumping a traceback and dying. Credential and missing-database
|
||||
errors are NOT transient and fail immediately.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
delay = _BACKOFF_INITIAL_SECONDS
|
||||
attempt = 0
|
||||
last_exc: BaseException | None = None
|
||||
|
||||
while True:
|
||||
attempt += 1
|
||||
try:
|
||||
asyncio.run(_probe(db_url))
|
||||
except Exception as exc: # broad by design — classified just below
|
||||
sqlstate = _fatal_sqlstate(exc)
|
||||
if sqlstate is not None:
|
||||
# Wrong password / missing database: waiting cannot fix it.
|
||||
raise
|
||||
last_exc = exc
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
logger.warning(
|
||||
"Database not ready (attempt %d, %.0fs budget left): %s — "
|
||||
"retrying in %.1fs",
|
||||
attempt, remaining, _describe(exc), min(delay, remaining),
|
||||
)
|
||||
time.sleep(min(delay, remaining))
|
||||
delay = min(delay * 2, _BACKOFF_MAX_SECONDS)
|
||||
else:
|
||||
if attempt > 1:
|
||||
logger.info("Database ready after %d attempt(s)", attempt)
|
||||
return
|
||||
|
||||
detail = f" Last error: {_describe(last_exc)}" if last_exc is not None else ""
|
||||
raise RuntimeError(
|
||||
f"Database did not become available within {timeout_seconds:.0f}s "
|
||||
f"({attempt} attempts).{detail}"
|
||||
) from last_exc
|
||||
|
||||
|
||||
class DatabaseUnavailable(Exception):
|
||||
"""The database could not be reached while serving a request."""
|
||||
|
||||
|
||||
async def ensure_database_reachable(engine: AsyncEngine) -> None:
|
||||
"""Check out one pooled connection, retrying briefly, or raise.
|
||||
|
||||
Startup readiness (wait_for_database) does not help once the app is already
|
||||
serving: if the database goes away underneath a running Steward, the next
|
||||
request needs a connection, the pooled ones are dead, and establishing a new
|
||||
one fails — which is how a login attempt turned into an opaque 500.
|
||||
|
||||
pool_pre_ping already makes recovery automatic *once the database is back*.
|
||||
What it cannot do is wait: while the container is genuinely down, its
|
||||
replacement connect fails too. So retry briefly here to ride out a restart,
|
||||
then give up and let the caller render an honest "database unavailable"
|
||||
page rather than a generic error.
|
||||
|
||||
The budget is deliberately short — a person is waiting on this request, and
|
||||
a page that hangs for half a minute is worse than one that says plainly
|
||||
what is wrong and offers a retry.
|
||||
"""
|
||||
last_exc: BaseException | None = None
|
||||
for attempt, delay in enumerate(_REQUEST_RETRY_DELAYS + (None,)):
|
||||
try:
|
||||
async with engine.connect():
|
||||
if attempt:
|
||||
logger.info(
|
||||
"Database reachable again after %d retry attempt(s)", attempt)
|
||||
return
|
||||
except (SQLAlchemyError, OSError) as exc:
|
||||
# Bad credentials / missing database will never resolve by waiting,
|
||||
# and they are not what this guard is for — let them surface.
|
||||
if _fatal_sqlstate(exc) is not None:
|
||||
raise
|
||||
last_exc = exc
|
||||
if delay is not None:
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
raise DatabaseUnavailable(_describe(last_exc)) from last_exc
|
||||
|
||||
|
||||
async def _probe(db_url: str) -> None:
|
||||
"""Open one throwaway connection and round-trip a trivial query.
|
||||
|
||||
Uses its own engine with NullPool: this runs before the app engine exists,
|
||||
and a probe connection must never be left in a pool for real work to reuse.
|
||||
"""
|
||||
engine = create_async_engine(db_url, echo=False, poolclass=NullPool)
|
||||
try:
|
||||
async with engine.connect() as conn:
|
||||
await conn.execute(text("SELECT 1"))
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
+233
-14
@@ -1,15 +1,17 @@
|
||||
from __future__ import annotations
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from quart import Blueprint, render_template, request, redirect, url_for, current_app, session
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import and_, case, select, func
|
||||
from steward.ansible import executor, sources as ansible_src
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.core.audit import log_audit
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.monitors import Monitor, MonitorResult
|
||||
from steward.models.users import UserRole
|
||||
from steward.hosts import service as host_svc
|
||||
from steward.core.status import _last_n_results, _uptime_by_monitor, _heartbeat_title
|
||||
|
||||
hosts_bp = Blueprint("hosts", __name__, url_prefix="/hosts")
|
||||
|
||||
@@ -21,18 +23,235 @@ def _ansible_source_names() -> list[str]:
|
||||
return []
|
||||
|
||||
|
||||
async def _compute_uptime(db) -> dict[str, dict]:
|
||||
"""Per-host uptime % for 24h/7d/30d, aggregated over the host's monitors.
|
||||
|
||||
Returns {host_id: {"24h": float|None, "7d": float|None, "30d": float|None}}.
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
cutoff_30d = now - timedelta(days=30)
|
||||
cutoff_7d = now - timedelta(days=7)
|
||||
cutoff_24h = now - timedelta(hours=24)
|
||||
|
||||
result = await db.execute(
|
||||
select(
|
||||
Monitor.host_id,
|
||||
func.count(MonitorResult.id).label("total_30d"),
|
||||
func.sum(case((MonitorResult.is_up.is_(True), 1), else_=0)).label("up_30d"),
|
||||
func.sum(case((MonitorResult.checked_at >= cutoff_7d, 1), else_=0)).label("total_7d"),
|
||||
func.sum(case(
|
||||
(and_(MonitorResult.checked_at >= cutoff_7d, MonitorResult.is_up.is_(True)), 1),
|
||||
else_=0,
|
||||
)).label("up_7d"),
|
||||
func.sum(case((MonitorResult.checked_at >= cutoff_24h, 1), else_=0)).label("total_24h"),
|
||||
func.sum(case(
|
||||
(and_(MonitorResult.checked_at >= cutoff_24h, MonitorResult.is_up.is_(True)), 1),
|
||||
else_=0,
|
||||
)).label("up_24h"),
|
||||
)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(Monitor.host_id.isnot(None), MonitorResult.checked_at >= cutoff_30d)
|
||||
.group_by(Monitor.host_id)
|
||||
)
|
||||
|
||||
def _pct(up, total):
|
||||
return round(float(up) / float(total) * 100, 2) if total else None
|
||||
|
||||
stats: dict[str, dict] = {}
|
||||
for row in result:
|
||||
stats[row.host_id] = {
|
||||
"24h": _pct(row.up_24h, row.total_24h),
|
||||
"7d": _pct(row.up_7d, row.total_7d),
|
||||
"30d": _pct(row.up_30d, row.total_30d),
|
||||
}
|
||||
return stats
|
||||
|
||||
|
||||
async def _host_status(db, host_ids: list[str]) -> dict[str, dict]:
|
||||
"""Per-host status rollup from the latest result of each linked monitor.
|
||||
|
||||
{host_id: {"state": "up"|"down"|"pending", "latency_ms": float|None,
|
||||
"count": int}}. State is down if ANY linked monitor's latest
|
||||
result is down (worst-case), up if at least one is up, else pending.
|
||||
"""
|
||||
if not host_ids:
|
||||
return {}
|
||||
rn = func.row_number().over(
|
||||
partition_by=MonitorResult.monitor_id, order_by=MonitorResult.checked_at.desc()
|
||||
).label("rn")
|
||||
subq = (
|
||||
select(MonitorResult.id, rn)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(Monitor.host_id.in_(host_ids), Monitor.enabled.is_(True))
|
||||
.subquery()
|
||||
)
|
||||
rows = (await db.execute(
|
||||
select(MonitorResult, Monitor.host_id)
|
||||
.join(subq, MonitorResult.id == subq.c.id)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(subq.c.rn == 1)
|
||||
)).all()
|
||||
out: dict[str, dict] = {}
|
||||
for res, host_id in rows:
|
||||
d = out.setdefault(host_id, {"state": "pending", "latency_ms": None, "count": 0})
|
||||
d["count"] += 1
|
||||
if not res.is_up:
|
||||
d["state"] = "down"
|
||||
elif d["state"] != "down":
|
||||
d["state"] = "up"
|
||||
if res.is_up and res.response_ms is not None:
|
||||
d["latency_ms"] = (res.response_ms if d["latency_ms"] is None
|
||||
else min(d["latency_ms"], res.response_ms))
|
||||
return out
|
||||
|
||||
|
||||
async def _host_monitors(db, host_id: str) -> list[dict]:
|
||||
"""Display rows for the monitors linked to one host (host hub section)."""
|
||||
monitors = (await db.execute(
|
||||
select(Monitor).where(Monitor.host_id == host_id).order_by(Monitor.name)
|
||||
)).scalars().all()
|
||||
ids = [m.id for m in monitors]
|
||||
recent = await _last_n_results(db, ids)
|
||||
uptime = await _uptime_by_monitor(db, ids)
|
||||
data = []
|
||||
for m in monitors:
|
||||
mrows = recent.get(m.id, [])
|
||||
latest = mrows[-1] if mrows else None
|
||||
data.append({
|
||||
"monitor": m, "latest": latest,
|
||||
"heartbeat": [{"state": "up" if r.is_up else "down",
|
||||
"title": _heartbeat_title(m.type, r)} for r in mrows],
|
||||
"uptime_pct": uptime.get(m.id, {}).get("24h"),
|
||||
})
|
||||
return data
|
||||
|
||||
|
||||
async def _agent_metrics_by_host(db, host_names: list[str]) -> dict[str, dict]:
|
||||
"""Latest agent cpu/mem per host name from the generic PluginMetric table.
|
||||
|
||||
Read directly (PluginMetric is a core model) so the Hosts list can show an
|
||||
at-a-glance agent column without importing the host_agent plugin.
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
if not host_names:
|
||||
return {}
|
||||
wanted = ("cpu_pct", "mem_used_pct")
|
||||
subq = (
|
||||
select(
|
||||
PluginMetric.resource_name, PluginMetric.metric_name,
|
||||
func.max(PluginMetric.recorded_at).label("m"),
|
||||
)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.metric_name.in_(wanted),
|
||||
PluginMetric.resource_name.in_(host_names),
|
||||
)
|
||||
.group_by(PluginMetric.resource_name, PluginMetric.metric_name)
|
||||
).subquery()
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric).join(
|
||||
subq,
|
||||
(PluginMetric.resource_name == subq.c.resource_name)
|
||||
& (PluginMetric.metric_name == subq.c.metric_name)
|
||||
& (PluginMetric.recorded_at == subq.c.m),
|
||||
)
|
||||
)).scalars().all()
|
||||
out: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
out.setdefault(r.resource_name, {})[r.metric_name] = r.value
|
||||
return out
|
||||
|
||||
|
||||
async def _agent_overview_by_host(db, host_names: list[str]) -> dict[str, dict]:
|
||||
"""Per-host agent glance (cpu/mem/root-disk + freshness) from PluginMetric.
|
||||
|
||||
Core-safe (no host_agent import). Freshness compares the latest host-level
|
||||
sample against the plugin's stale window (config), defaulting to 180s.
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
if not host_names:
|
||||
return {}
|
||||
now = datetime.now(timezone.utc)
|
||||
stale_after = int(
|
||||
(current_app.config.get("PLUGINS", {}).get("host_agent", {}) or {})
|
||||
.get("stale_after_seconds", 180))
|
||||
resources = list(host_names) + [h + ":/" for h in host_names]
|
||||
subq = (
|
||||
select(
|
||||
PluginMetric.resource_name, PluginMetric.metric_name,
|
||||
func.max(PluginMetric.recorded_at).label("m"),
|
||||
)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.resource_name.in_(resources),
|
||||
PluginMetric.metric_name.in_(("cpu_pct", "mem_used_pct", "disk_used_pct")),
|
||||
)
|
||||
.group_by(PluginMetric.resource_name, PluginMetric.metric_name)
|
||||
).subquery()
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric).join(
|
||||
subq,
|
||||
(PluginMetric.resource_name == subq.c.resource_name)
|
||||
& (PluginMetric.metric_name == subq.c.metric_name)
|
||||
& (PluginMetric.recorded_at == subq.c.m),
|
||||
).where(PluginMetric.source_module == "host_agent")
|
||||
)).scalars().all()
|
||||
out: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
if r.resource_name.endswith(":/") and r.metric_name == "disk_used_pct":
|
||||
out.setdefault(r.resource_name[:-2], {})["disk_root"] = r.value
|
||||
elif r.resource_name in host_names:
|
||||
d = out.setdefault(r.resource_name, {})
|
||||
d[r.metric_name] = r.value
|
||||
if d.get("_ts") is None or r.recorded_at > d["_ts"]:
|
||||
d["_ts"] = r.recorded_at
|
||||
for d in out.values():
|
||||
ts = d.pop("_ts", None)
|
||||
d["fresh"] = bool(ts and (now - ts).total_seconds() <= stale_after)
|
||||
return out
|
||||
|
||||
|
||||
async def _agent_cpu_sparks_by_host(db, host_names: list[str], hours: int = 1) -> dict[str, str]:
|
||||
"""{host_name: inline-SVG cpu sparkline} over the last `hours` (core-safe).
|
||||
|
||||
A tiny at-a-glance trend in each row — the host-view sparkline, on the widget.
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.core.status import sparkline_svg
|
||||
if not host_names:
|
||||
return {}
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric.resource_name, PluginMetric.value, PluginMetric.recorded_at)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.metric_name == "cpu_pct",
|
||||
PluginMetric.resource_name.in_(host_names),
|
||||
PluginMetric.recorded_at >= cutoff,
|
||||
)
|
||||
.order_by(PluginMetric.recorded_at)
|
||||
)).all()
|
||||
by_host: dict[str, list[float]] = {}
|
||||
for r in rows:
|
||||
by_host.setdefault(r.resource_name, []).append(r.value)
|
||||
# Amber line to read as "CPU"; only draw when there are a couple of points.
|
||||
return {
|
||||
name: sparkline_svg(vals[-40:], width=64, height=18, stroke="#c8a840")
|
||||
for name, vals in by_host.items() if len(vals) >= 2
|
||||
}
|
||||
|
||||
|
||||
@hosts_bp.get("/overview/widget")
|
||||
@require_role(UserRole.viewer)
|
||||
async def overview_widget():
|
||||
"""Dashboard widget: unified per-host monitor + agent glance, linking to the hub."""
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
hosts = (await db.execute(select(Host).order_by(Host.name))).scalars().all()
|
||||
host_status = await host_svc.host_status(db, [h.id for h in hosts])
|
||||
uptime = await host_svc.compute_uptime(db)
|
||||
host_status = await _host_status(db, [h.id for h in hosts])
|
||||
uptime = await _compute_uptime(db)
|
||||
host_names = [h.name for h in hosts]
|
||||
agent = await host_svc.agent_overview_by_host(
|
||||
db, host_names, host_svc.agent_stale_after_seconds(current_app.config))
|
||||
cpu_sparks = await host_svc.agent_cpu_sparks_by_host(db, host_names)
|
||||
agent = await _agent_overview_by_host(db, host_names)
|
||||
cpu_sparks = await _agent_cpu_sparks_by_host(db, host_names)
|
||||
|
||||
return await render_template(
|
||||
"hosts/overview_widget.html",
|
||||
@@ -47,9 +266,9 @@ async def list_hosts():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
result = await db.execute(select(Host).order_by(Host.name))
|
||||
hosts = result.scalars().all()
|
||||
host_status = await host_svc.host_status(db, [h.id for h in hosts])
|
||||
uptime = await host_svc.compute_uptime(db)
|
||||
agent_metrics = await host_svc.agent_metrics_by_host(db, [h.name for h in hosts])
|
||||
host_status = await _host_status(db, [h.id for h in hosts])
|
||||
uptime = await _compute_uptime(db)
|
||||
agent_metrics = await _agent_metrics_by_host(db, [h.name for h in hosts])
|
||||
|
||||
return await render_template(
|
||||
"hosts/list.html",
|
||||
@@ -72,8 +291,8 @@ async def host_detail(host_id: str):
|
||||
select(Host).where(Host.id == host_id))).scalar_one_or_none()
|
||||
if host is None:
|
||||
return "Not found", 404
|
||||
monitors = await host_svc.host_monitors(db, host_id)
|
||||
uptime = (await host_svc.compute_uptime(db)).get(host_id)
|
||||
monitors = await _host_monitors(db, host_id)
|
||||
uptime = (await _compute_uptime(db)).get(host_id)
|
||||
linked_target = (await db.execute(
|
||||
select(AnsibleTarget).where(AnsibleTarget.host_id == host_id)
|
||||
.options(selectinload(AnsibleTarget.groups)))).scalar_one_or_none()
|
||||
@@ -281,7 +500,7 @@ async def uptime_page():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
result = await db.execute(select(Host).order_by(Host.name))
|
||||
hosts = result.scalars().all()
|
||||
uptime = await host_svc.compute_uptime(db)
|
||||
uptime = await _compute_uptime(db)
|
||||
return await render_template("hosts/uptime.html", hosts=hosts, uptime=uptime)
|
||||
|
||||
|
||||
@@ -292,7 +511,7 @@ async def uptime_widget():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
result = await db.execute(select(Host).order_by(Host.name))
|
||||
hosts = result.scalars().all()
|
||||
uptime = await host_svc.compute_uptime(db)
|
||||
uptime = await _compute_uptime(db)
|
||||
return await render_template(
|
||||
"hosts/uptime_widget.html",
|
||||
hosts=hosts,
|
||||
|
||||
@@ -1,247 +0,0 @@
|
||||
"""Host queries shared by the Hosts pages, dashboard widgets and MCP tools.
|
||||
|
||||
Every function takes an open session and returns plain data; none reads
|
||||
Quart's request context, so the web routes and the /mcp tools (which run
|
||||
outside Quart's pipeline) call the same code.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import and_, case, func, select
|
||||
|
||||
from steward.core.status import _heartbeat_title, _last_n_results, _uptime_by_monitor
|
||||
from steward.models.monitors import Monitor, MonitorResult
|
||||
|
||||
|
||||
# Host-level agent metrics read for the at-a-glance views (plus the root
|
||||
# mount's disk_used_pct, read from the "<host>:/" sub-resource).
|
||||
AGENT_GLANCE_METRICS = ("cpu_pct", "mem_used_pct", "disk_used_pct", "disk_used_pct_worst")
|
||||
|
||||
|
||||
def agent_stale_after_seconds(config) -> int:
|
||||
"""The host_agent plugin's freshness window from app config (default 180s)."""
|
||||
return int(((config.get("PLUGINS") or {}).get("host_agent") or {})
|
||||
.get("stale_after_seconds", 180))
|
||||
|
||||
|
||||
async def compute_uptime(db) -> dict[str, dict]:
|
||||
"""Per-host uptime % for 24h/7d/30d, aggregated over the host's monitors.
|
||||
|
||||
Returns {host_id: {"24h": float|None, "7d": float|None, "30d": float|None}}.
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
cutoff_30d = now - timedelta(days=30)
|
||||
cutoff_7d = now - timedelta(days=7)
|
||||
cutoff_24h = now - timedelta(hours=24)
|
||||
|
||||
result = await db.execute(
|
||||
select(
|
||||
Monitor.host_id,
|
||||
func.count(MonitorResult.id).label("total_30d"),
|
||||
func.sum(case((MonitorResult.is_up.is_(True), 1), else_=0)).label("up_30d"),
|
||||
func.sum(case((MonitorResult.checked_at >= cutoff_7d, 1), else_=0)).label("total_7d"),
|
||||
func.sum(case(
|
||||
(and_(MonitorResult.checked_at >= cutoff_7d, MonitorResult.is_up.is_(True)), 1),
|
||||
else_=0,
|
||||
)).label("up_7d"),
|
||||
func.sum(case((MonitorResult.checked_at >= cutoff_24h, 1), else_=0)).label("total_24h"),
|
||||
func.sum(case(
|
||||
(and_(MonitorResult.checked_at >= cutoff_24h, MonitorResult.is_up.is_(True)), 1),
|
||||
else_=0,
|
||||
)).label("up_24h"),
|
||||
)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(Monitor.host_id.isnot(None), MonitorResult.checked_at >= cutoff_30d)
|
||||
.group_by(Monitor.host_id)
|
||||
)
|
||||
|
||||
def _pct(up, total):
|
||||
return round(float(up) / float(total) * 100, 2) if total else None
|
||||
|
||||
stats: dict[str, dict] = {}
|
||||
for row in result:
|
||||
stats[row.host_id] = {
|
||||
"24h": _pct(row.up_24h, row.total_24h),
|
||||
"7d": _pct(row.up_7d, row.total_7d),
|
||||
"30d": _pct(row.up_30d, row.total_30d),
|
||||
}
|
||||
return stats
|
||||
|
||||
|
||||
async def host_status(db, host_ids: list[str]) -> dict[str, dict]:
|
||||
"""Per-host status rollup from the latest result of each linked monitor.
|
||||
|
||||
{host_id: {"state": "up"|"down"|"pending", "latency_ms": float|None,
|
||||
"count": int}}. State is down if ANY linked monitor's latest
|
||||
result is down (worst-case), up if at least one is up, else pending.
|
||||
"""
|
||||
if not host_ids:
|
||||
return {}
|
||||
rn = func.row_number().over(
|
||||
partition_by=MonitorResult.monitor_id, order_by=MonitorResult.checked_at.desc()
|
||||
).label("rn")
|
||||
subq = (
|
||||
select(MonitorResult.id, rn)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(Monitor.host_id.in_(host_ids), Monitor.enabled.is_(True))
|
||||
.subquery()
|
||||
)
|
||||
rows = (await db.execute(
|
||||
select(MonitorResult, Monitor.host_id)
|
||||
.join(subq, MonitorResult.id == subq.c.id)
|
||||
.join(Monitor, Monitor.id == MonitorResult.monitor_id)
|
||||
.where(subq.c.rn == 1)
|
||||
)).all()
|
||||
out: dict[str, dict] = {}
|
||||
for res, host_id in rows:
|
||||
d = out.setdefault(host_id, {"state": "pending", "latency_ms": None, "count": 0})
|
||||
d["count"] += 1
|
||||
if not res.is_up:
|
||||
d["state"] = "down"
|
||||
elif d["state"] != "down":
|
||||
d["state"] = "up"
|
||||
if res.is_up and res.response_ms is not None:
|
||||
d["latency_ms"] = (res.response_ms if d["latency_ms"] is None
|
||||
else min(d["latency_ms"], res.response_ms))
|
||||
return out
|
||||
|
||||
|
||||
async def host_monitors(db, host_id: str) -> list[dict]:
|
||||
"""Display rows for the monitors linked to one host (host hub section)."""
|
||||
monitors = (await db.execute(
|
||||
select(Monitor).where(Monitor.host_id == host_id).order_by(Monitor.name)
|
||||
)).scalars().all()
|
||||
ids = [m.id for m in monitors]
|
||||
recent = await _last_n_results(db, ids)
|
||||
uptime = await _uptime_by_monitor(db, ids)
|
||||
data = []
|
||||
for m in monitors:
|
||||
mrows = recent.get(m.id, [])
|
||||
latest = mrows[-1] if mrows else None
|
||||
data.append({
|
||||
"monitor": m, "latest": latest,
|
||||
"heartbeat": [{"state": "up" if r.is_up else "down",
|
||||
"title": _heartbeat_title(m.type, r)} for r in mrows],
|
||||
"uptime_pct": uptime.get(m.id, {}).get("24h"),
|
||||
})
|
||||
return data
|
||||
|
||||
|
||||
async def agent_metrics_by_host(db, host_names: list[str]) -> dict[str, dict]:
|
||||
"""Latest agent cpu/mem per host name from the generic PluginMetric table.
|
||||
|
||||
Read directly (PluginMetric is a core model) so the Hosts list can show an
|
||||
at-a-glance agent column without importing the host_agent plugin.
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
if not host_names:
|
||||
return {}
|
||||
wanted = ("cpu_pct", "mem_used_pct")
|
||||
subq = (
|
||||
select(
|
||||
PluginMetric.resource_name, PluginMetric.metric_name,
|
||||
func.max(PluginMetric.recorded_at).label("m"),
|
||||
)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.metric_name.in_(wanted),
|
||||
PluginMetric.resource_name.in_(host_names),
|
||||
)
|
||||
.group_by(PluginMetric.resource_name, PluginMetric.metric_name)
|
||||
).subquery()
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric).join(
|
||||
subq,
|
||||
(PluginMetric.resource_name == subq.c.resource_name)
|
||||
& (PluginMetric.metric_name == subq.c.metric_name)
|
||||
& (PluginMetric.recorded_at == subq.c.m),
|
||||
)
|
||||
)).scalars().all()
|
||||
out: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
out.setdefault(r.resource_name, {})[r.metric_name] = r.value
|
||||
return out
|
||||
|
||||
|
||||
async def agent_overview_by_host(
|
||||
db, host_names: list[str], stale_after_seconds: int = 180,
|
||||
) -> dict[str, dict]:
|
||||
"""Per-host agent glance (cpu/mem/disk + freshness) from PluginMetric.
|
||||
|
||||
{host_name: {cpu_pct, mem_used_pct, disk_used_pct_worst, disk_root,
|
||||
last_at, fresh}} for each host with any agent samples.
|
||||
|
||||
Core-safe (no host_agent import). Freshness compares the latest host-level
|
||||
sample against the host_agent plugin's stale window (stale_after_seconds;
|
||||
callers pass agent_stale_after_seconds(app.config)).
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
if not host_names:
|
||||
return {}
|
||||
now = datetime.now(timezone.utc)
|
||||
stale_after = int(stale_after_seconds)
|
||||
resources = list(host_names) + [h + ":/" for h in host_names]
|
||||
subq = (
|
||||
select(
|
||||
PluginMetric.resource_name, PluginMetric.metric_name,
|
||||
func.max(PluginMetric.recorded_at).label("m"),
|
||||
)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.resource_name.in_(resources),
|
||||
PluginMetric.metric_name.in_(AGENT_GLANCE_METRICS),
|
||||
)
|
||||
.group_by(PluginMetric.resource_name, PluginMetric.metric_name)
|
||||
).subquery()
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric).join(
|
||||
subq,
|
||||
(PluginMetric.resource_name == subq.c.resource_name)
|
||||
& (PluginMetric.metric_name == subq.c.metric_name)
|
||||
& (PluginMetric.recorded_at == subq.c.m),
|
||||
).where(PluginMetric.source_module == "host_agent")
|
||||
)).scalars().all()
|
||||
out: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
if r.resource_name.endswith(":/") and r.metric_name == "disk_used_pct":
|
||||
out.setdefault(r.resource_name[:-2], {})["disk_root"] = r.value
|
||||
elif r.resource_name in host_names:
|
||||
d = out.setdefault(r.resource_name, {})
|
||||
d[r.metric_name] = r.value
|
||||
if d.get("_ts") is None or r.recorded_at > d["_ts"]:
|
||||
d["_ts"] = r.recorded_at
|
||||
for d in out.values():
|
||||
ts = d.pop("_ts", None)
|
||||
d["last_at"] = ts
|
||||
d["fresh"] = bool(ts and (now - ts).total_seconds() <= stale_after)
|
||||
return out
|
||||
|
||||
|
||||
async def agent_cpu_sparks_by_host(db, host_names: list[str], hours: int = 1) -> dict[str, str]:
|
||||
"""{host_name: inline-SVG cpu sparkline} over the last `hours` (core-safe).
|
||||
|
||||
A tiny at-a-glance trend in each row — the host-view sparkline, on the widget.
|
||||
"""
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.core.status import sparkline_svg
|
||||
if not host_names:
|
||||
return {}
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
|
||||
rows = (await db.execute(
|
||||
select(PluginMetric.resource_name, PluginMetric.value, PluginMetric.recorded_at)
|
||||
.where(
|
||||
PluginMetric.source_module == "host_agent",
|
||||
PluginMetric.metric_name == "cpu_pct",
|
||||
PluginMetric.resource_name.in_(host_names),
|
||||
PluginMetric.recorded_at >= cutoff,
|
||||
)
|
||||
.order_by(PluginMetric.recorded_at)
|
||||
)).all()
|
||||
by_host: dict[str, list[float]] = {}
|
||||
for r in rows:
|
||||
by_host.setdefault(r.resource_name, []).append(r.value)
|
||||
# Amber line to read as "CPU"; only draw when there are a couple of points.
|
||||
return {
|
||||
name: sparkline_svg(vals[-40:], width=64, height=18, stroke="#c8a840")
|
||||
for name, vals in by_host.items() if len(vals) >= 2
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
"""The MCP endpoint (/mcp): Steward as a server an LLM client can inspect and drive."""
|
||||
from .server import mount_mcp
|
||||
|
||||
__all__ = ["mount_mcp"]
|
||||
@@ -1,50 +0,0 @@
|
||||
"""Per-request MCP context: who is calling, and which app they are calling.
|
||||
|
||||
The /mcp transport bypasses Quart's request pipeline, so neither `session`
|
||||
nor `current_app` exists inside a tool. The ASGI auth wrapper in server.py
|
||||
sets these from the resolved bearer token before dispatching; tools read them
|
||||
through `current_caller()` and `mcp_app()`.
|
||||
|
||||
Neither falls back to a default: a tool reached without an authenticated
|
||||
caller is a bug in the transport, and it raises rather than acting for nobody.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from contextvars import ContextVar
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from quart import Quart
|
||||
from steward.core.api_tokens import Caller
|
||||
|
||||
_caller_ctx: ContextVar["Caller | None"] = ContextVar("steward_mcp_caller", default=None)
|
||||
_app_ctx: ContextVar["Quart | None"] = ContextVar("steward_mcp_app", default=None)
|
||||
# The scheme and host the caller reached /mcp on, e.g. "https://steward.example".
|
||||
_base_url_ctx: ContextVar[str] = ContextVar("steward_mcp_base_url", default="")
|
||||
|
||||
|
||||
def current_caller() -> "Caller":
|
||||
caller = _caller_ctx.get()
|
||||
if caller is None:
|
||||
raise RuntimeError("no MCP caller in context: the request was not bearer-authenticated")
|
||||
return caller
|
||||
|
||||
|
||||
def current_caller_or_none() -> "Caller | None":
|
||||
return _caller_ctx.get()
|
||||
|
||||
|
||||
def mcp_app() -> "Quart":
|
||||
app = _app_ctx.get()
|
||||
if app is None:
|
||||
raise RuntimeError("no Steward app in MCP context")
|
||||
return app
|
||||
|
||||
|
||||
def public_url() -> str:
|
||||
"""Steward's externally reachable base URL, as the web side's
|
||||
public_base_url() gives it: Settings → General → Public base URL when set,
|
||||
else the scheme and host this MCP request arrived on; "" when neither is
|
||||
known."""
|
||||
app = _app_ctx.get()
|
||||
configured = (app.config.get("PUBLIC_BASE_URL") or "").strip() if app is not None else ""
|
||||
return (configured or _base_url_ctx.get()).rstrip("/")
|
||||
@@ -1,302 +0,0 @@
|
||||
"""Steward's MCP server: the StewardMCPServer instance and its mount at /mcp.
|
||||
|
||||
The /mcp path is dispatched to the MCP SDK's streamable-HTTP ASGI app BEFORE
|
||||
Quart sees it (wrapping `app.asgi_app`), so Quart's session, before_request
|
||||
hooks and error pages do not apply. This module supplies the equivalents:
|
||||
|
||||
- bearer auth: a personal API token resolves to a Caller (user + effective
|
||||
role), set in a ContextVar for the tools — 401 otherwise;
|
||||
- the database gate: the same ride-out-a-restart probe the web pipeline uses,
|
||||
answering 503 rather than failing deep in a tool;
|
||||
- role gating, default-deny: every tool is registered with a minimum role and
|
||||
a tool the caller's role doesn't reach is neither listed nor callable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import difflib
|
||||
import functools
|
||||
import json
|
||||
import logging
|
||||
|
||||
from mcp.server.mcpserver import MCPServer
|
||||
from mcp.server.mcpserver.exceptions import ToolError, UnexpectedToolError
|
||||
from mcp.server.transport_security import TransportSecuritySettings
|
||||
from mcp.types import ToolAnnotations
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from steward.auth.middleware import role_meets
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from . import _context
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MCP_PATH = "/mcp"
|
||||
|
||||
# Claude Code injects only the first ~2,048 characters of a server's
|
||||
# instructions; keep this well under (test_instructions_fit). The workflow
|
||||
# across tools lives here; each tool's own contract lives in its docstring.
|
||||
INSTRUCTIONS = """\
|
||||
Steward is a self-hosted monitoring and infrastructure hub: hosts, uptime
|
||||
monitors, host-agent metrics, Docker containers and logs, alerts, and Ansible
|
||||
automation. You act as the user who created your token, at that token's role.
|
||||
|
||||
Start broad, then narrow: get_overview says what is wrong right now; follow
|
||||
its ids into get_host / get_monitor / get_container, then metrics and logs.
|
||||
List tools are bounded: a response with `truncated: true` says which argument
|
||||
fetches more. Times are UTC ISO-8601; units are in field names (_pct, _ms,
|
||||
_bytes).
|
||||
|
||||
Changes are audited under the user's name. run_playbook is a dry run (Ansible
|
||||
check mode) unless you pass check=false; dry-run first and show the user the
|
||||
diff before applying. If a tool refuses for lack of role, whoami says what
|
||||
this token can do.
|
||||
"""
|
||||
|
||||
_DB_UNAVAILABLE = (
|
||||
"Steward is running but can't reach its database right now, so nothing was "
|
||||
"read or changed. This is usually a database restart; retry in a minute."
|
||||
)
|
||||
|
||||
|
||||
def _is_db_failure(exc: BaseException | None) -> bool:
|
||||
seen: set[int] = set()
|
||||
while exc is not None and id(exc) not in seen:
|
||||
seen.add(id(exc))
|
||||
if isinstance(exc, (SQLAlchemyError, OSError, TimeoutError)):
|
||||
return True
|
||||
exc = exc.__cause__ or exc.__context__
|
||||
return False
|
||||
|
||||
|
||||
class StewardMCPServer(MCPServer):
|
||||
"""MCPServer with per-tool minimum roles, plugin-scoped tools and strict args."""
|
||||
|
||||
def __init__(self, app, **kwargs):
|
||||
super().__init__("steward", instructions=INSTRUCTIONS.strip(), **kwargs)
|
||||
self._steward_app = app
|
||||
self.tool_roles: dict[str, UserRole] = {}
|
||||
self.tool_plugins: dict[str, str] = {}
|
||||
# plugin name -> its `mcp_overview(db)`: findings it adds to get_overview.
|
||||
self.overview_sources: dict[str, object] = {}
|
||||
|
||||
# ── registration ─────────────────────────────────────────────────────────
|
||||
def register_tool(self, fn, *, role: UserRole, read_only: bool, plugin: str | None = None) -> None:
|
||||
"""The only way a tool reaches the server: name, minimum role, effect.
|
||||
|
||||
`role` and `read_only` are required — there is no default for a
|
||||
mutating tool to inherit by omission.
|
||||
"""
|
||||
name = fn.__name__
|
||||
if name in self.tool_roles:
|
||||
raise ValueError(f"MCP tool {name!r} is already registered")
|
||||
if not isinstance(role, UserRole):
|
||||
raise TypeError(f"MCP tool {name!r}: role must be a UserRole, got {role!r}")
|
||||
self.add_tool(
|
||||
fn, name=name,
|
||||
annotations=ToolAnnotations(read_only_hint=read_only,
|
||||
destructive_hint=None if read_only else True),
|
||||
)
|
||||
self.tool_roles[name] = role
|
||||
if plugin:
|
||||
self.tool_plugins[name] = plugin
|
||||
|
||||
def register_plugin(self, name: str, module) -> None:
|
||||
"""Add a plugin's tools via its optional `register_mcp_tools(register)`,
|
||||
and its optional `mcp_overview(db)` as a get_overview findings source.
|
||||
|
||||
Idempotent per plugin (hot-reload drops its previous tools first) and
|
||||
tolerant: a raising hook is logged and leaves the plugin with no tools,
|
||||
never a broken server.
|
||||
"""
|
||||
self._drop_plugin_tools(name)
|
||||
self.overview_sources.pop(name, None)
|
||||
overview = getattr(module, "mcp_overview", None)
|
||||
if callable(overview):
|
||||
self.overview_sources[name] = overview
|
||||
hook = getattr(module, "register_mcp_tools", None)
|
||||
if hook is None:
|
||||
return
|
||||
try:
|
||||
hook(functools.partial(self.register_tool, plugin=name))
|
||||
except Exception:
|
||||
# All or nothing: a hook that fails halfway leaves no partial set.
|
||||
self._drop_plugin_tools(name)
|
||||
logger.exception("Plugin %r: register_mcp_tools() raised; its MCP tools are unavailable", name)
|
||||
|
||||
def _drop_plugin_tools(self, name: str) -> None:
|
||||
for tool_name in [t for t, p in self.tool_plugins.items() if p == name]:
|
||||
self.remove_tool(tool_name)
|
||||
self.tool_roles.pop(tool_name, None)
|
||||
self.tool_plugins.pop(tool_name, None)
|
||||
|
||||
# ── gating ───────────────────────────────────────────────────────────────
|
||||
def plugin_enabled(self, plugin: str) -> bool:
|
||||
cfg = (self._steward_app.config.get("PLUGINS") or {}).get(plugin)
|
||||
return isinstance(cfg, dict) and bool(cfg.get("enabled"))
|
||||
|
||||
def refusal(self, name: str, caller) -> str | None:
|
||||
"""Why `caller` may not use tool `name`, or None if they may."""
|
||||
required = self.tool_roles.get(name)
|
||||
if required is None:
|
||||
return f"There is no Steward tool named {name!r}."
|
||||
plugin = self.tool_plugins.get(name)
|
||||
if plugin and not self.plugin_enabled(plugin):
|
||||
return (f"{name} belongs to the {plugin} plugin, which is disabled in this "
|
||||
"Steward. An admin can enable it under Settings → Plugins.")
|
||||
if caller is None:
|
||||
return "This request is not authenticated."
|
||||
if not role_meets(caller.role, required):
|
||||
return (f"{name} needs the {required.value} role; this connection acts as "
|
||||
f"{caller.role.value}. A token's role is capped at its user's role — "
|
||||
"create a token with a higher role on Steward's AI access page if "
|
||||
"your account allows it.")
|
||||
return None
|
||||
|
||||
async def list_tools(self):
|
||||
caller = _context.current_caller_or_none()
|
||||
tools = await super().list_tools()
|
||||
return [t for t in tools if self.refusal(t.name, caller) is None]
|
||||
|
||||
async def call_tool(self, name, arguments, context=None):
|
||||
refusal = self.refusal(name, _context.current_caller_or_none())
|
||||
if refusal:
|
||||
raise ToolError(refusal)
|
||||
|
||||
# Reject undeclared arguments: pydantic would silently drop a misnamed
|
||||
# one and run the tool with that field's default.
|
||||
tool = self._tool_manager.get_tool(name)
|
||||
declared = set((tool.parameters or {}).get("properties", {})) if tool else set()
|
||||
unknown = sorted(set(arguments or {}) - declared)
|
||||
if unknown:
|
||||
hints = []
|
||||
for arg in unknown:
|
||||
close = difflib.get_close_matches(arg, sorted(declared), n=1)
|
||||
hints.append(f"'{arg}'" + (f" (did you mean '{close[0]}'?)" if close else ""))
|
||||
raise ToolError(
|
||||
f"{name} does not accept {', '.join(hints)}. It accepts: "
|
||||
f"{', '.join(sorted(declared)) or 'no arguments'}. Nothing was run."
|
||||
)
|
||||
|
||||
try:
|
||||
return await super().call_tool(name, arguments, context)
|
||||
except UnexpectedToolError as exc:
|
||||
cause = exc.__cause__
|
||||
# ValueError is how a Steward tool refuses: what, why, what instead.
|
||||
if isinstance(cause, ValueError):
|
||||
raise ToolError(f"{name}: {cause}") from cause
|
||||
if _is_db_failure(cause):
|
||||
logger.warning("MCP tool %s hit a database failure: %r", name, cause)
|
||||
raise ToolError(_DB_UNAVAILABLE) from cause
|
||||
raise
|
||||
|
||||
|
||||
def build_mcp_server(app) -> StewardMCPServer:
|
||||
from .tools import register_all
|
||||
mcp = StewardMCPServer(app)
|
||||
register_all(mcp.register_tool)
|
||||
return mcp
|
||||
|
||||
|
||||
async def _send_json(send, status: int, payload: dict, extra_headers=()) -> None:
|
||||
await send({
|
||||
"type": "http.response.start",
|
||||
"status": status,
|
||||
"headers": [(b"content-type", b"application/json"), *extra_headers],
|
||||
})
|
||||
await send({"type": "http.response.body", "body": json.dumps(payload).encode()})
|
||||
|
||||
|
||||
def mount_mcp(app) -> StewardMCPServer:
|
||||
"""Mount the MCP endpoint at /mcp and return the server.
|
||||
|
||||
stateless_http: every request stands alone on its bearer token. The
|
||||
stateful default strands Claude Code after a redeploy — it reconnects with
|
||||
a session id the new process never issued and won't re-initialize on the
|
||||
404 — so stateless is what makes a restart invisible to clients.
|
||||
|
||||
DNS-rebinding protection is off: its default allow-list is localhost only,
|
||||
so any deployment behind a reverse proxy would get 421s. That protection
|
||||
guards a browser rebinding DNS onto a localhost server; here the bearer
|
||||
token is the security boundary.
|
||||
"""
|
||||
# Looked up through their modules per request (not bound here) so a test
|
||||
# can stand in for the token store or the database probe.
|
||||
from steward import database
|
||||
from steward.core import api_tokens
|
||||
|
||||
from steward.core.plugin_manager import get_loaded_plugin_modules
|
||||
|
||||
mcp = build_mcp_server(app)
|
||||
for name, module in sorted(get_loaded_plugin_modules().items()):
|
||||
mcp.register_plugin(name, module)
|
||||
mcp_asgi = mcp.streamable_http_app(
|
||||
streamable_http_path=MCP_PATH,
|
||||
stateless_http=True,
|
||||
transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False),
|
||||
)
|
||||
app.mcp_server = mcp
|
||||
|
||||
# The SDK's session manager owns a task group that must be running before
|
||||
# it serves; host it in Quart's serving lifecycle.
|
||||
@app.before_serving
|
||||
async def _start_mcp() -> None:
|
||||
cm = mcp.session_manager.run()
|
||||
await cm.__aenter__()
|
||||
app._mcp_session_cm = cm
|
||||
|
||||
@app.after_serving
|
||||
async def _stop_mcp() -> None:
|
||||
cm = getattr(app, "_mcp_session_cm", None)
|
||||
if cm is not None:
|
||||
await cm.__aexit__(None, None, None)
|
||||
|
||||
async def handle_mcp(scope, receive, send):
|
||||
headers = {k.decode("latin-1").lower(): v.decode("latin-1")
|
||||
for k, v in scope.get("headers", [])}
|
||||
auth = headers.get("authorization", "")
|
||||
raw = auth[7:].strip() if auth[:7].lower() == "bearer " else ""
|
||||
|
||||
engine = getattr(app, "_db_engine", None)
|
||||
try:
|
||||
if engine is not None:
|
||||
await database.ensure_database_reachable(engine)
|
||||
caller = await api_tokens.resolve_token(app, raw)
|
||||
except (database.DatabaseUnavailable, SQLAlchemyError, OSError) as exc:
|
||||
logger.error("MCP request refused, database unreachable: %r", exc)
|
||||
await _send_json(send, 503, {"error": "database_unavailable", "message": _DB_UNAVAILABLE})
|
||||
return
|
||||
|
||||
if caller is None:
|
||||
await _send_json(
|
||||
send, 401,
|
||||
{"error": "unauthorized",
|
||||
"message": "Send a Steward API token as 'Authorization: Bearer stw_…'. "
|
||||
"Create one on Steward's AI access page."},
|
||||
extra_headers=[(b"www-authenticate", b'Bearer realm="steward-mcp"')],
|
||||
)
|
||||
return
|
||||
|
||||
caller_token = _context._caller_ctx.set(caller)
|
||||
app_token = _context._app_ctx.set(app)
|
||||
host = headers.get("host", "")
|
||||
url_token = _context._base_url_ctx.set(f"{scope.get('scheme', 'http')}://{host}" if host else "")
|
||||
try:
|
||||
await mcp_asgi(scope, receive, send)
|
||||
finally:
|
||||
_context._caller_ctx.reset(caller_token)
|
||||
_context._app_ctx.reset(app_token)
|
||||
_context._base_url_ctx.reset(url_token)
|
||||
|
||||
original_asgi = app.asgi_app
|
||||
|
||||
async def dispatch(scope, receive, send):
|
||||
if scope["type"] == "http":
|
||||
path = scope.get("path", "")
|
||||
if path == MCP_PATH or path.startswith(MCP_PATH + "/"):
|
||||
# Path passes through untouched: the SDK app routes on /mcp itself.
|
||||
return await handle_mcp(scope, receive, send)
|
||||
return await original_asgi(scope, receive, send)
|
||||
|
||||
app.asgi_app = dispatch
|
||||
return mcp
|
||||
@@ -1,22 +0,0 @@
|
||||
"""Core MCP tool modules.
|
||||
|
||||
Each module defines plain `async def` tools at module level (tests call them
|
||||
directly) and a `register(register_tool)` that lists them with their minimum
|
||||
role. `register_tool(fn, role=..., read_only=...)` takes both as required
|
||||
keywords: a tool with no declared role is unreachable, and there is no default
|
||||
read_only for a mutating tool to inherit by forgetting.
|
||||
|
||||
A tool's docstring IS its contract with the agent — it ships as the tool's
|
||||
description. Say what it is for, when to reach for it over its siblings, what
|
||||
each argument's empty value means, and what comes back.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from . import alerts, audit, automation, hosts, metrics, monitors, overview, session
|
||||
|
||||
_MODULES = (session, overview, hosts, monitors, alerts, metrics, audit, automation)
|
||||
|
||||
|
||||
def register_all(register_tool) -> None:
|
||||
for module in _MODULES:
|
||||
module.register(register_tool)
|
||||
@@ -1,24 +0,0 @@
|
||||
"""Audit trail for changes made through MCP.
|
||||
|
||||
Every mutating tool records its change with `audit_mcp`, under the calling
|
||||
user's id and the username "<user> via MCP" (an operator ruling on the API
|
||||
Surface), so the audit log separates what a person did from what an agent
|
||||
did on their token. The token id goes into the detail.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from steward.core.audit import log_audit
|
||||
from steward.mcp._context import current_caller, mcp_app
|
||||
|
||||
VIA = " via MCP"
|
||||
|
||||
|
||||
def mcp_username(username: str) -> str:
|
||||
return (username + VIA)[:64]
|
||||
|
||||
|
||||
async def audit_mcp(action: str, entity_type: str | None = None, entity_id: str | None = None,
|
||||
detail: dict | None = None) -> None:
|
||||
caller = current_caller()
|
||||
await log_audit(mcp_app(), caller.user_id, mcp_username(caller.username), action,
|
||||
entity_type, entity_id, {**(detail or {}), "token_id": caller.token_id})
|
||||
@@ -1,149 +0,0 @@
|
||||
"""Response shaping every tool shares: the conventions INSTRUCTIONS promise.
|
||||
|
||||
- Times are ISO-8601 UTC with a `Z`, and anything with a "when" also carries
|
||||
`age_s` so the agent never does clock arithmetic.
|
||||
- Every list is bounded: a tool returns `total`, `truncated`, and when cut, a
|
||||
`more` line naming the argument that fetches the rest. Nothing is cut
|
||||
silently.
|
||||
- A time series comes back as a summary plus a downsampled series, never raw
|
||||
rows.
|
||||
- An entity is looked up by id or by name, and a miss raises ValueError naming
|
||||
the tool that lists the valid choices (the server turns it into a readable
|
||||
ToolError).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import difflib
|
||||
import math
|
||||
from datetime import datetime, timezone
|
||||
from typing import Callable, Iterable, TypeVar
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from steward.core.time_range import RANGES
|
||||
from steward.mcp._context import mcp_app
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
SERIES_POINTS = 60 # max points in any returned series
|
||||
|
||||
|
||||
def db_session():
|
||||
"""A session on the calling Steward's database; use as `async with db_session() as db`."""
|
||||
return mcp_app().db_sessionmaker()
|
||||
|
||||
|
||||
def now_utc() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _aware(dt: datetime) -> datetime:
|
||||
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def iso(dt: datetime | None) -> str | None:
|
||||
if dt is None:
|
||||
return None
|
||||
return _aware(dt).astimezone(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def age_s(dt: datetime | None, now: datetime | None = None) -> int | None:
|
||||
if dt is None:
|
||||
return None
|
||||
return max(0, int(((now or now_utc()) - _aware(dt)).total_seconds()))
|
||||
|
||||
|
||||
def rnd(value, digits: int = 2):
|
||||
return None if value is None else round(float(value), digits)
|
||||
|
||||
|
||||
def clamp_limit(limit: int, default: int, maximum: int) -> int:
|
||||
"""`limit` <= 0 means the default; anything above `maximum` is capped."""
|
||||
if not limit or limit <= 0:
|
||||
return default
|
||||
return min(int(limit), maximum)
|
||||
|
||||
|
||||
def bounded(items: list, limit: int, *, key: str = "items", more: str = "") -> dict:
|
||||
"""`{key: items[:limit], total, truncated}`, plus `more` when something was cut."""
|
||||
total = len(items)
|
||||
out = {key: items[:limit], "total": total, "truncated": total > limit}
|
||||
if total > limit:
|
||||
out["more"] = f"Showing {limit} of {total}. " + (more or "Pass a larger `limit`.")
|
||||
return out
|
||||
|
||||
|
||||
def lookback(range_: str) -> tuple[datetime, str]:
|
||||
"""A lookback window as (since, key). Unlike core parse_range, an unknown key
|
||||
is refused rather than quietly read as 24h: the agent should learn its typo."""
|
||||
key = (range_ or "24h").strip()
|
||||
if key not in RANGES:
|
||||
raise ValueError(f"range {range_!r} is not one of {', '.join(RANGES)}.")
|
||||
return now_utc() - RANGES[key], key
|
||||
|
||||
|
||||
def downsample(points: list[tuple[datetime, float]], n: int = SERIES_POINTS) -> list[tuple[datetime, float]]:
|
||||
"""Average consecutive points into at most n, each stamped with its first time."""
|
||||
if len(points) <= n:
|
||||
return points
|
||||
size = math.ceil(len(points) / n)
|
||||
out = []
|
||||
for i in range(0, len(points), size):
|
||||
chunk = points[i:i + size]
|
||||
out.append((chunk[0][0], sum(v for _, v in chunk) / len(chunk)))
|
||||
return out
|
||||
|
||||
|
||||
def summarize(points: list[tuple[datetime, float]], n: int = SERIES_POINTS) -> dict:
|
||||
"""min/avg/max/p95/last over oldest-first (time, value) points, plus a ≤n series."""
|
||||
if not points:
|
||||
return {"points": 0, "series": []}
|
||||
values = [v for _, v in points]
|
||||
ordered = sorted(values)
|
||||
p95 = ordered[max(0, math.ceil(0.95 * len(ordered)) - 1)]
|
||||
return {
|
||||
"points": len(values),
|
||||
"min": rnd(ordered[0]),
|
||||
"avg": rnd(sum(values) / len(values)),
|
||||
"max": rnd(ordered[-1]),
|
||||
"p95": rnd(p95),
|
||||
"last": rnd(values[-1]),
|
||||
"last_at": iso(points[-1][0]),
|
||||
"series": [[iso(t), rnd(v)] for t, v in downsample(points, n)],
|
||||
}
|
||||
|
||||
|
||||
def pick(want: str, rows: Iterable[T], *, kind: str, list_tool: str,
|
||||
name: Callable[[T], str] = lambda r: r.name,
|
||||
ident: Callable[[T], str] = lambda r: r.id) -> T:
|
||||
"""The row whose id is `want`, else the one whose name matches (case-insensitive)."""
|
||||
rows = list(rows)
|
||||
want = (want or "").strip()
|
||||
if not want:
|
||||
raise ValueError(f"pass a {kind} name or id; {list_tool} lists them.")
|
||||
for r in rows:
|
||||
if ident(r) == want:
|
||||
return r
|
||||
same = [r for r in rows if (name(r) or "").lower() == want.lower()]
|
||||
if len(same) == 1:
|
||||
return same[0]
|
||||
if same:
|
||||
ids = ", ".join(f"{ident(r)}" for r in same)
|
||||
raise ValueError(f"{len(same)} {kind}s are named {want!r}; pass one of these ids instead: {ids}.")
|
||||
close = difflib.get_close_matches(want.lower(), [(name(r) or "").lower() for r in rows], n=3, cutoff=0.5)
|
||||
hint = f" Did you mean {', '.join(repr(c) for c in close)}?" if close else ""
|
||||
raise ValueError(f"no {kind} named or with id {want!r}.{hint} {list_tool} lists them.")
|
||||
|
||||
|
||||
def redact_url(url: str) -> str:
|
||||
"""Drop a password from a URL's userinfo; the rest stays readable."""
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
except ValueError:
|
||||
return url
|
||||
if parts.password is None:
|
||||
return url
|
||||
host = parts.hostname or ""
|
||||
if parts.port:
|
||||
host = f"{host}:{parts.port}"
|
||||
netloc = f"{parts.username}:***@{host}" if parts.username else f"***@{host}"
|
||||
return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment))
|
||||
@@ -1,398 +0,0 @@
|
||||
"""Alert tools: what is firing, what fired before, the rules, and maintenance windows."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.mcp._context import current_caller
|
||||
from steward.models.alerts import AlertEvent, AlertOperator, AlertRule, AlertState, MaintenanceWindow
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
from ._audit import audit_mcp
|
||||
from ._shape import (
|
||||
age_s, bounded, clamp_limit, db_session, iso, lookback, now_utc, pick, rnd,
|
||||
)
|
||||
|
||||
|
||||
def _condition(rule: AlertRule) -> str:
|
||||
n = rule.consecutive_failures_required or 1
|
||||
tail = f" for {n} consecutive checks" if n > 1 else ""
|
||||
return f"{rule.metric_name} {rule.operator.value} {rule.threshold:g}{tail}"
|
||||
|
||||
|
||||
def _action(rule: AlertRule) -> dict | None:
|
||||
"""The playbook a rule runs when it fires (alerts/routes.py _parse_ansible_action)."""
|
||||
a = rule.ansible_action
|
||||
if not a:
|
||||
return None
|
||||
return {k: a.get(k) for k in ("source", "playbook", "inventory", "limit", "tags",
|
||||
"extra_vars", "check") if a.get(k) is not None}
|
||||
|
||||
|
||||
async def _usernames(db, ids) -> dict[str, str]:
|
||||
ids = {i for i in ids if i}
|
||||
if not ids:
|
||||
return {}
|
||||
return dict((await db.execute(select(User.id, User.username).where(User.id.in_(ids)))).all())
|
||||
|
||||
|
||||
async def list_alerts() -> dict:
|
||||
"""Alerts that need attention now: firing, pending, or acknowledged but still breaching.
|
||||
|
||||
pending: the condition holds but not yet for the rule's required number of
|
||||
consecutive checks. firing: it has. acknowledged: someone has seen it and
|
||||
it still holds. Each carries the rule's condition, the resource and metric
|
||||
it watches, and `since` (its latest state change). Firing, unacknowledged
|
||||
alerts come first. An alert that cleared is not here; get_alert_history
|
||||
shows the transitions.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
active = await alert_svc.active_alerts(db)
|
||||
names = await _usernames(db, [s.acknowledged_by for s, _ in active])
|
||||
order = {"firing": 0, "pending": 1, "acknowledged": 2}
|
||||
rows = [
|
||||
{
|
||||
"rule_id": rule.id,
|
||||
"rule_name": rule.name,
|
||||
"state": state.state.value,
|
||||
"source_module": rule.source_module,
|
||||
"resource": rule.resource_name,
|
||||
"metric": rule.metric_name,
|
||||
"condition": _condition(rule),
|
||||
"since": iso(state.last_transition_at),
|
||||
"age_s": age_s(state.last_transition_at),
|
||||
"acknowledged_by": names.get(state.acknowledged_by) if state.acknowledged_by else None,
|
||||
"acknowledged_at": iso(state.acknowledged_at),
|
||||
}
|
||||
for state, rule in active
|
||||
]
|
||||
rows.sort(key=lambda r: (order.get(r["state"], 3), -(r["age_s"] or 0)))
|
||||
return {"alerts": rows, "total": len(rows)}
|
||||
|
||||
|
||||
async def get_alert_history(rule: str = "", range: str = "7d", limit: int = 50) -> dict:
|
||||
"""State transitions of alert rules, newest first: when each fired, cleared or was acked.
|
||||
|
||||
rule: one rule's name or id (list_alert_rules shows them); empty for every rule.
|
||||
range: "1h", "6h", "24h", "7d" (default) or "30d".
|
||||
limit: default 50, max 500.
|
||||
|
||||
Each transition carries the metric value that caused it and whether the
|
||||
email/webhook notification went out (with the error if not), plus the id
|
||||
of any Ansible run the rule's action triggered.
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
limit = clamp_limit(limit, 50, 500)
|
||||
async with db_session() as db:
|
||||
rules = await alert_svc.list_rules(db)
|
||||
stmt = (select(AlertEvent, AlertRule)
|
||||
.join(AlertRule, AlertEvent.rule_id == AlertRule.id)
|
||||
.where(AlertEvent.transitioned_at >= since)
|
||||
.order_by(AlertEvent.transitioned_at.desc())
|
||||
.limit(limit + 1))
|
||||
chosen = None
|
||||
if rule:
|
||||
chosen = pick(rule, rules, kind="alert rule", list_tool="list_alert_rules")
|
||||
stmt = stmt.where(AlertEvent.rule_id == chosen.id)
|
||||
rows = (await db.execute(stmt)).all()
|
||||
events = [
|
||||
{
|
||||
"rule_id": r.id,
|
||||
"rule_name": r.name,
|
||||
"from_state": e.from_state,
|
||||
"to_state": e.to_state,
|
||||
"metric_value": rnd(e.metric_value),
|
||||
"at": iso(e.transitioned_at),
|
||||
"age_s": age_s(e.transitioned_at),
|
||||
"email_sent": e.email_sent,
|
||||
"email_error": e.email_error,
|
||||
"webhook_sent": e.webhook_sent,
|
||||
"webhook_error": e.webhook_error,
|
||||
"ansible_run_id": e.ansible_run_id,
|
||||
}
|
||||
for e, r in rows
|
||||
]
|
||||
out = {"range": key, "rule": chosen.name if chosen else None}
|
||||
out.update(bounded(events, limit, key="events",
|
||||
more="Pass a larger `limit` (max 500), a shorter `range`, or one `rule`."))
|
||||
if out["truncated"]:
|
||||
# Fetched limit+1 rows to detect the cut; the true total isn't known.
|
||||
out["total"] = None
|
||||
out["more"] = f"More than {limit} transitions in range. Pass a larger `limit` (max 500), a shorter `range`, or one `rule`."
|
||||
return out
|
||||
|
||||
|
||||
async def list_alert_rules(enabled_only: bool = False, limit: int = 200) -> dict:
|
||||
"""Every alert rule: what it watches, its condition, current state and automatic action.
|
||||
|
||||
enabled_only: true to leave out disabled rules.
|
||||
limit: default 200, max 500.
|
||||
|
||||
`state` is the rule's current state: "inactive" (condition not met),
|
||||
"pending", "firing", "acknowledged", "resolved" (cleared since it last
|
||||
fired), or null if it has never been evaluated. `ansible_action` is the
|
||||
playbook the rule runs when it fires, if any (`check: true` = dry run).
|
||||
"""
|
||||
limit = clamp_limit(limit, 200, 500)
|
||||
async with db_session() as db:
|
||||
rules = await alert_svc.list_rules(db)
|
||||
states = {s.rule_id: s for s in (await db.execute(select(AlertState))).scalars()}
|
||||
if enabled_only:
|
||||
rules = [r for r in rules if r.enabled]
|
||||
rows = [
|
||||
{
|
||||
"rule_id": r.id,
|
||||
"rule_name": r.name,
|
||||
"enabled": r.enabled,
|
||||
"source_module": r.source_module,
|
||||
"resource": r.resource_name,
|
||||
"metric": r.metric_name,
|
||||
"condition": _condition(r),
|
||||
"state": states[r.id].state.value if r.id in states else None,
|
||||
"state_since": iso(states[r.id].last_transition_at) if r.id in states else None,
|
||||
"ansible_action": _action(r),
|
||||
}
|
||||
for r in rules
|
||||
]
|
||||
return bounded(rows, limit, key="rules")
|
||||
|
||||
|
||||
async def list_maintenance_windows(include_past: bool = False, limit: int = 50) -> dict:
|
||||
"""Maintenance windows, during which matching alerts are held back from notifying.
|
||||
|
||||
include_past: true to include windows that have ended (default: current
|
||||
and upcoming only).
|
||||
limit: default 50, max 500.
|
||||
|
||||
`scope` is what a window silences: a source module and/or one resource, or
|
||||
everything when both are null. `active` is true while it is in effect.
|
||||
"""
|
||||
limit = clamp_limit(limit, 50, 500)
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
windows = await alert_svc.list_maintenance_windows(db)
|
||||
rows = [
|
||||
{
|
||||
"window_id": w.id,
|
||||
"name": w.name,
|
||||
"start_at": iso(w.start_at),
|
||||
"end_at": iso(w.end_at),
|
||||
"active": w.start_at <= now <= w.end_at,
|
||||
"scope": {"source_module": w.scope_source, "resource": w.scope_resource},
|
||||
}
|
||||
for w in windows
|
||||
if include_past or w.end_at >= now
|
||||
]
|
||||
return bounded(rows, limit, key="windows")
|
||||
|
||||
|
||||
# ── changes ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def get_alert_rule_options(source_module: str = "") -> dict:
|
||||
"""What an alert rule can watch: the choices create_alert_rule accepts.
|
||||
|
||||
source_module: empty lists every source and its metrics; a source's name
|
||||
also lists its resources (hosts, containers, monitors… whatever that
|
||||
source reports under).
|
||||
|
||||
A monitor reports under its type (icmp/tcp/dns/http) with its name as the
|
||||
resource: is_up (1 up, 0 down) and response_ms.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
if not source_module:
|
||||
return {"sources": {src: await alert_svc.metrics_for(db, src)
|
||||
for src in alert_svc.SOURCE_MODULES},
|
||||
"operators": [o.value for o in AlertOperator]}
|
||||
if source_module not in alert_svc.METRIC_CATALOG:
|
||||
raise ValueError(f"source_module {source_module!r} is not one of "
|
||||
f"{', '.join(alert_svc.SOURCE_MODULES)}.")
|
||||
return {"source_module": source_module,
|
||||
"metrics": await alert_svc.metrics_for(db, source_module),
|
||||
"resources": await alert_svc.resources_for(db, source_module),
|
||||
"operators": [o.value for o in AlertOperator]}
|
||||
|
||||
|
||||
async def _rule(db, rule: str) -> AlertRule:
|
||||
return pick(rule, await alert_svc.list_rules(db), kind="alert rule", list_tool="list_alert_rules")
|
||||
|
||||
|
||||
def _rule_row(r: AlertRule) -> dict:
|
||||
return {"rule_id": r.id, "rule_name": r.name, "enabled": r.enabled,
|
||||
"source_module": r.source_module, "resource": r.resource_name,
|
||||
"metric": r.metric_name, "condition": _condition(r), "ansible_action": _action(r)}
|
||||
|
||||
|
||||
async def create_alert_rule(name: str, source_module: str, resource: str, metric: str,
|
||||
operator: str, threshold: float, consecutive_checks: int = 1,
|
||||
enabled: bool = True) -> dict:
|
||||
"""Add an alert rule: notify when `metric` of `resource` crosses `threshold`.
|
||||
|
||||
source_module, resource, metric: get_alert_rule_options lists the valid
|
||||
choices; a resource must have reported (or, for a monitor, exist).
|
||||
operator: ">", "<", ">=", "<=", "==" or "!=".
|
||||
consecutive_checks: how many checks in a row must breach before it fires
|
||||
(default 1); higher values ride out a single blip.
|
||||
|
||||
Notifications go to the channels set in Steward's Settings. The playbook a
|
||||
rule runs when it fires is set separately (set_alert_action, admin).
|
||||
Recorded in the audit log as "<you> via MCP".
|
||||
"""
|
||||
caller = current_caller()
|
||||
async with db_session() as db, db.begin():
|
||||
fields = await alert_svc.rule_fields(
|
||||
db, name=name, source_module=source_module, resource_name=resource,
|
||||
metric_name=metric, operator=operator, threshold=threshold,
|
||||
consecutive_failures_required=consecutive_checks)
|
||||
r = await alert_svc.add_rule(db, fields, created_by=caller.user_id, enabled=enabled)
|
||||
await audit_mcp("alert_rule.created", "alert_rule", r.name,
|
||||
{"rule_id": r.id, "source": r.source_module, "resource": r.resource_name,
|
||||
"metric": r.metric_name})
|
||||
return _rule_row(r)
|
||||
|
||||
|
||||
async def update_alert_rule(rule: str, name: str | None = None, source_module: str | None = None,
|
||||
resource: str | None = None, metric: str | None = None,
|
||||
operator: str | None = None, threshold: float | None = None,
|
||||
consecutive_checks: int | None = None,
|
||||
enabled: bool | None = None) -> dict:
|
||||
"""Change an alert rule. Only the arguments you pass change; the rest stay.
|
||||
|
||||
rule: its name or id (list_alert_rules). Pass enabled=false to silence it
|
||||
without deleting it. The changed rule is validated as create_alert_rule
|
||||
validates a new one. Recorded in the audit log.
|
||||
"""
|
||||
async with db_session() as db, db.begin():
|
||||
r = await _rule(db, rule)
|
||||
fields = await alert_svc.rule_fields(
|
||||
db,
|
||||
name=r.name if name is None else name,
|
||||
source_module=r.source_module if source_module is None else source_module,
|
||||
resource_name=r.resource_name if resource is None else resource,
|
||||
metric_name=r.metric_name if metric is None else metric,
|
||||
operator=r.operator.value if operator is None else operator,
|
||||
threshold=r.threshold if threshold is None else threshold,
|
||||
consecutive_failures_required=(r.consecutive_failures_required
|
||||
if consecutive_checks is None else consecutive_checks))
|
||||
for k, v in fields.items():
|
||||
setattr(r, k, v)
|
||||
if enabled is not None:
|
||||
r.enabled = bool(enabled)
|
||||
changed = sorted(k for k, v in {"name": name, "source_module": source_module,
|
||||
"resource": resource, "metric": metric, "operator": operator,
|
||||
"threshold": threshold, "consecutive_checks": consecutive_checks,
|
||||
"enabled": enabled}.items() if v is not None)
|
||||
await audit_mcp("alert_rule.updated", "alert_rule", r.name, {"rule_id": r.id, "changed": changed})
|
||||
return _rule_row(r)
|
||||
|
||||
|
||||
async def delete_alert_rule(rule: str) -> dict:
|
||||
"""Delete an alert rule and its history. To silence it but keep it,
|
||||
update_alert_rule(enabled=false). Admin only; recorded in the audit log."""
|
||||
async with db_session() as db, db.begin():
|
||||
r = await _rule(db, rule)
|
||||
await db.delete(r)
|
||||
await audit_mcp("alert_rule.deleted", "alert_rule", r.name, {"rule_id": r.id})
|
||||
return {"deleted": r.name, "rule_id": r.id}
|
||||
|
||||
|
||||
async def acknowledge_alert(rule: str) -> dict:
|
||||
"""Acknowledge a firing alert: you have seen it. It stays "acknowledged"
|
||||
while the condition holds and resolves when it clears.
|
||||
|
||||
rule: the firing rule's name or id (list_alerts). Recorded in the audit log.
|
||||
"""
|
||||
caller = current_caller()
|
||||
async with db_session() as db, db.begin():
|
||||
r = await _rule(db, rule)
|
||||
try:
|
||||
state = await alert_svc.acknowledge(db, r.id, caller.user_id)
|
||||
except LookupError:
|
||||
raise ValueError(f"{r.name} has never been evaluated, so there is nothing to "
|
||||
"acknowledge.") from None
|
||||
await audit_mcp("alert_rule.acknowledged", "alert_rule", r.name, {"rule_id": r.id})
|
||||
return {"rule_id": r.id, "rule_name": r.name, "state": state.state.value,
|
||||
"acknowledged_at": iso(state.acknowledged_at)}
|
||||
|
||||
|
||||
def _when(value: str, what: str) -> datetime:
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip().replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
raise ValueError(f"{what} {value!r} is not an ISO-8601 time, e.g. 2026-10-06T22:00:00Z.") from None
|
||||
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _window_row(w: MaintenanceWindow, now: datetime) -> dict:
|
||||
return {"window_id": w.id, "name": w.name, "start_at": iso(w.start_at), "end_at": iso(w.end_at),
|
||||
"active": w.start_at <= now <= w.end_at,
|
||||
"scope": {"source_module": w.scope_source, "resource": w.scope_resource}}
|
||||
|
||||
|
||||
async def create_maintenance_window(name: str, duration_minutes: int = 0, end_at: str = "",
|
||||
start_at: str = "", source_module: str = "",
|
||||
resource: str = "") -> dict:
|
||||
"""Hold back alert notifications while you work on something.
|
||||
|
||||
Give the end as duration_minutes OR end_at (ISO-8601; a time with no zone
|
||||
is UTC). start_at defaults to now.
|
||||
source_module, resource: limit the window to one source, or to one
|
||||
resource of it; both empty silences everything.
|
||||
|
||||
Alerts still change state during a window, so list_alerts shows them;
|
||||
only notifications are held. Recorded in the audit log.
|
||||
"""
|
||||
now = now_utc()
|
||||
start = _when(start_at, "start_at") if (start_at or "").strip() else now
|
||||
if bool(duration_minutes) == bool((end_at or "").strip()):
|
||||
raise ValueError("give exactly one of duration_minutes or end_at.")
|
||||
if duration_minutes:
|
||||
if duration_minutes < 0:
|
||||
raise ValueError("duration_minutes must be positive.")
|
||||
end = start + timedelta(minutes=duration_minutes)
|
||||
else:
|
||||
end = _when(end_at, "end_at")
|
||||
fields = alert_svc.window_fields(name=name, start_at=start, end_at=end,
|
||||
scope_source=source_module, scope_resource=resource)
|
||||
w = MaintenanceWindow(created_by=current_caller().user_id, **fields)
|
||||
async with db_session() as db, db.begin():
|
||||
db.add(w)
|
||||
await audit_mcp("maintenance_window.created", "maintenance_window", w.name,
|
||||
{"window_id": w.id, "scope": alert_svc.window_scope(w),
|
||||
"start_at": iso(w.start_at), "end_at": iso(w.end_at)})
|
||||
return _window_row(w, now)
|
||||
|
||||
|
||||
async def end_maintenance_window(window: str) -> dict:
|
||||
"""End a maintenance window now, so notifications resume.
|
||||
|
||||
window: its name or id (list_maintenance_windows). A window in progress
|
||||
ends now and stays on record; one that has not started yet is removed.
|
||||
Recorded in the audit log.
|
||||
"""
|
||||
now = now_utc()
|
||||
async with db_session() as db, db.begin():
|
||||
windows = await alert_svc.list_maintenance_windows(db)
|
||||
w = pick(window, windows, kind="maintenance window", list_tool="list_maintenance_windows")
|
||||
if w.end_at <= now:
|
||||
raise ValueError(f"{w.name} already ended at {iso(w.end_at)}.")
|
||||
if w.start_at > now:
|
||||
await db.delete(w)
|
||||
outcome = "cancelled"
|
||||
else:
|
||||
w.end_at = now
|
||||
outcome = "ended"
|
||||
await audit_mcp(f"maintenance_window.{outcome}", "maintenance_window", w.name,
|
||||
{"window_id": w.id})
|
||||
return {**_window_row(w, now), "outcome": outcome}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
for fn in (list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows,
|
||||
get_alert_rule_options):
|
||||
register_tool(fn, role=UserRole.viewer, read_only=True)
|
||||
for fn in (create_alert_rule, update_alert_rule, acknowledge_alert,
|
||||
create_maintenance_window, end_maintenance_window):
|
||||
register_tool(fn, role=UserRole.operator, read_only=False)
|
||||
register_tool(delete_alert_rule, role=UserRole.admin, read_only=False)
|
||||
@@ -1,74 +0,0 @@
|
||||
"""The audit log: who changed what in Steward, including changes made through MCP."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from sqlalchemy import or_, select
|
||||
|
||||
from steward.models.audit import AuditEvent
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from ._audit import mcp_username
|
||||
from ._shape import age_s, bounded, clamp_limit, db_session, iso, lookback
|
||||
|
||||
|
||||
async def list_audit_events(action: str = "", user: str = "", entity_type: str = "",
|
||||
range: str = "7d", limit: int = 50) -> dict:
|
||||
"""Changes recorded in Steward's audit log, newest first.
|
||||
|
||||
action: a prefix of the action name, e.g. "monitor." or "ansible.run";
|
||||
empty for all.
|
||||
user: a username, exactly; "system" for automated changes. Includes the
|
||||
changes that user made through MCP, which are recorded as "<user> via MCP".
|
||||
entity_type: e.g. "monitor", "host", "alert_rule"; empty for all.
|
||||
range: "1h", "6h", "24h", "7d" (default) or "30d".
|
||||
limit: default 50, max 500.
|
||||
|
||||
Answers "what changed before this broke?": line an event's `at` up against
|
||||
a monitor's down_since or an alert's since.
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
limit = clamp_limit(limit, 50, 500)
|
||||
stmt = (select(AuditEvent).where(AuditEvent.timestamp >= since)
|
||||
.order_by(AuditEvent.timestamp.desc()).limit(limit + 1))
|
||||
if action:
|
||||
esc = action.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
stmt = stmt.where(AuditEvent.action.like(esc + "%", escape="\\"))
|
||||
if user:
|
||||
stmt = stmt.where(or_(AuditEvent.username == user,
|
||||
AuditEvent.username == mcp_username(user)))
|
||||
if entity_type:
|
||||
stmt = stmt.where(AuditEvent.entity_type == entity_type)
|
||||
async with db_session() as db:
|
||||
rows = list((await db.execute(stmt)).scalars())
|
||||
|
||||
def _detail(raw):
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except ValueError:
|
||||
return raw[:500]
|
||||
|
||||
events = [
|
||||
{
|
||||
"at": iso(e.timestamp),
|
||||
"age_s": age_s(e.timestamp),
|
||||
"user": e.username,
|
||||
"action": e.action,
|
||||
"entity_type": e.entity_type,
|
||||
"entity_id": e.entity_id,
|
||||
"detail": _detail(e.detail_json),
|
||||
}
|
||||
for e in rows
|
||||
]
|
||||
out = {"range": key, **bounded(events, limit, key="events")}
|
||||
if out["truncated"]:
|
||||
out["total"] = None
|
||||
out["more"] = (f"More than {limit} events in range. Pass a larger `limit` (max 500), "
|
||||
"a shorter `range`, or filter by `action`, `user` or `entity_type`.")
|
||||
return out
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(list_audit_events, role=UserRole.viewer, read_only=True)
|
||||
@@ -1,697 +0,0 @@
|
||||
"""Automation tools: playbooks, inventory, runs, schedules and alert actions.
|
||||
|
||||
Reads are viewer tools. Anything that starts a run or changes what runs is an
|
||||
operator tool (set_alert_action is admin, as on the alert rule form), and each
|
||||
of those writes an audit row as "<user> via MCP".
|
||||
|
||||
Playbook runs default to Ansible check mode (`--check --diff`): the agent sees
|
||||
what would change before anything does, and applies only by passing
|
||||
check=false. Variables the playbook marks secret reach the run but are never
|
||||
stored or echoed back.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from collections import deque
|
||||
from datetime import timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from steward.ansible import executor, sources as src_module
|
||||
from steward.ansible.inventory_gen import scope_labels
|
||||
from steward.ansible.runner import trigger_run
|
||||
from steward.mcp._context import current_caller, mcp_app, public_url
|
||||
from steward.models.alerts import AlertRule
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.ansible_inventory import AnsibleGroup, AnsibleTarget
|
||||
from steward.models.ansible_schedule import AnsibleSchedule
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
from ._audit import audit_mcp
|
||||
from ._shape import age_s, bounded, clamp_limit, db_session, iso, lookback, pick
|
||||
|
||||
ACTIVE = (AnsibleRunStatus.running, AnsibleRunStatus.queued)
|
||||
RECAP_KEYS = ("ok", "changed", "failed", "unreachable", "skipped")
|
||||
CONTENT_CHARS = 20000
|
||||
LINE_CHARS = 500
|
||||
WAIT_MAX_S = 120
|
||||
POLL_S = 2.0
|
||||
MASK = "<secret>"
|
||||
|
||||
|
||||
# ── sources and playbooks ────────────────────────────────────────────────────
|
||||
|
||||
def _sources() -> list[dict]:
|
||||
return src_module.get_sources(mcp_app().config.get("ANSIBLE", {}))
|
||||
|
||||
|
||||
def _source(name: str) -> dict:
|
||||
srcs = _sources()
|
||||
for s in srcs:
|
||||
if s["name"] == (name or "").strip():
|
||||
return s
|
||||
raise ValueError(f"no playbook source named {name!r}. Sources: "
|
||||
f"{', '.join(s['name'] for s in srcs)} (list_playbooks shows their playbooks).")
|
||||
|
||||
|
||||
def _playbook(source: dict, path: str) -> str:
|
||||
"""The playbook's content; refuses a path the source does not have."""
|
||||
path = (path or "").strip()
|
||||
if path not in src_module.discover_playbooks(source["path"]):
|
||||
raise ValueError(f"source {source['name']!r} has no playbook {path!r}; "
|
||||
f'list_playbooks(source="{source["name"]}") lists them.')
|
||||
content = src_module.read_playbook(source["path"], path)
|
||||
if content is None:
|
||||
raise ValueError(f"playbook {path!r} in {source['name']!r} could not be read.")
|
||||
return content
|
||||
|
||||
|
||||
def public_variables(variables: list[dict]) -> list[dict]:
|
||||
"""A playbook's run-time variables with the defaults of secret ones masked."""
|
||||
return [{**v, "default": MASK if v["secret"] and v["default"] not in ("", None) else v["default"]}
|
||||
for v in variables]
|
||||
|
||||
|
||||
# Variables Steward's own bundled playbooks take that Steward can answer itself.
|
||||
STEWARD_KNOWN_VARS = {"steward_url": public_url}
|
||||
|
||||
|
||||
def steward_defaults(source: dict, variables: list[dict], extra_vars: dict | None) -> tuple[dict, dict]:
|
||||
"""(extra_vars, filled): a built-in playbook's Steward-known variables
|
||||
(steward_url) filled in when the caller left them out or empty, as the
|
||||
web actions do. Other sources' playbooks are left as they are."""
|
||||
extra_vars = dict(extra_vars or {})
|
||||
if source.get("name") != src_module.BUILTIN_SOURCE_NAME:
|
||||
return extra_vars, {}
|
||||
filled = {}
|
||||
declared = {v["name"] for v in variables}
|
||||
for name, value in STEWARD_KNOWN_VARS.items():
|
||||
if name in declared and not str(extra_vars.get(name) or "").strip():
|
||||
known = value()
|
||||
if known:
|
||||
extra_vars[name] = filled[name] = known
|
||||
return extra_vars, filled
|
||||
|
||||
|
||||
def split_vars(variables: list[dict], extra_vars: dict | None) -> tuple[dict, dict]:
|
||||
"""(plain, secret): the values the run stores, and those it must not.
|
||||
|
||||
Refuses a vars_prompt variable that has no default and no value, since a
|
||||
run cannot stop to ask for it.
|
||||
"""
|
||||
extra_vars = dict(extra_vars or {})
|
||||
for k, v in extra_vars.items():
|
||||
if not isinstance(k, str) or not k.strip():
|
||||
raise ValueError("extra_vars keys must be non-empty variable names.")
|
||||
if not isinstance(v, (str, int, float, bool)):
|
||||
raise ValueError(f"extra_vars[{k!r}] must be a string, number or boolean.")
|
||||
secret_names = {v["name"] for v in variables if v["secret"]}
|
||||
missing = [v["name"] for v in variables if v["required"] and v["name"] not in extra_vars]
|
||||
if missing:
|
||||
raise ValueError(f"this playbook prompts for {', '.join(missing)} and has no default; "
|
||||
"pass a value in extra_vars. Nothing was run.")
|
||||
plain = {k: v for k, v in extra_vars.items() if k not in secret_names}
|
||||
secret = {k: v for k, v in extra_vars.items() if k in secret_names}
|
||||
return plain, secret
|
||||
|
||||
|
||||
def build_params(plain: dict, limit: str, tags: str, check: bool) -> dict:
|
||||
params: dict = {}
|
||||
if plain:
|
||||
params["extra_vars_map"] = plain
|
||||
if (limit or "").strip():
|
||||
params["limit"] = limit.strip()
|
||||
if (tags or "").strip():
|
||||
params["tags"] = tags.strip()
|
||||
if check:
|
||||
params["check"] = True
|
||||
return params
|
||||
|
||||
|
||||
async def _scope(db, source: dict, scope: str) -> tuple[str, str]:
|
||||
"""(stored scope, human description) from the forms the docstrings offer."""
|
||||
s = (scope or "all").strip()
|
||||
if s in ("all", "steward:all"):
|
||||
n = len((await db.execute(select(AnsibleTarget.id))).all())
|
||||
if not n:
|
||||
raise ValueError("Steward's inventory has no targets yet; add one under Ansible → "
|
||||
"Inventory, or pass scope=\"inventory:<file>\" for a repo inventory.")
|
||||
return "steward:all", f"all {n} inventory targets"
|
||||
kind, _, want = s.removeprefix("steward:").partition(":")
|
||||
if kind == "target":
|
||||
rows = (await db.execute(select(AnsibleTarget))).scalars()
|
||||
t = pick(want, rows, kind="target", list_tool="list_inventory")
|
||||
return f"steward:target:{t.id}", f"target {t.name} ({t.address})"
|
||||
if kind == "group":
|
||||
rows = (await db.execute(select(AnsibleGroup).options(selectinload(AnsibleGroup.targets)))).scalars()
|
||||
g = pick(want, rows, kind="group", list_tool="list_inventory")
|
||||
if not g.targets:
|
||||
raise ValueError(f"group {g.name!r} has no targets. Nothing was run.")
|
||||
return f"steward:group:{g.id}", f"group {g.name} ({len(g.targets)} targets)"
|
||||
if kind in ("inventory", "repo"):
|
||||
inv = want.rsplit(":", 1)[-1] if kind == "repo" else want
|
||||
found = src_module.discover_inventories(source["path"])
|
||||
if inv not in found:
|
||||
raise ValueError(f"source {source['name']!r} has no inventory file {inv!r}"
|
||||
+ (f"; it has {', '.join(found)}." if found else "; it has none."))
|
||||
return f"repo:{source['name']}:{inv}", f"{source['name']}/{inv}"
|
||||
raise ValueError(f"scope {scope!r} is not one of: all, target:<name or id>, "
|
||||
"group:<name or id>, inventory:<file>. list_inventory shows the choices.")
|
||||
|
||||
|
||||
async def list_playbooks(source: str = "", category: str = "", limit: int = 200) -> dict:
|
||||
"""The playbooks Steward can run, by source, with what each one says it does.
|
||||
|
||||
source: one source's name; empty for all. "steward-builtin" holds
|
||||
Steward's own maintenance playbooks, "steward-local" those written in
|
||||
Steward's editor; the rest come from configured directories or git repos.
|
||||
category: only playbooks whose `# steward:category:` matches; empty for all.
|
||||
limit: default 200, max 1000.
|
||||
|
||||
`confirm: true` marks a playbook its author flagged as destructive:
|
||||
run_playbook will apply it only with confirmed=true. get_playbook shows a
|
||||
playbook's variables and content.
|
||||
"""
|
||||
limit = clamp_limit(limit, 200, 1000)
|
||||
srcs = [_source(source)] if source else _sources()
|
||||
rows, out_sources = [], []
|
||||
for s in srcs:
|
||||
inventories = src_module.discover_inventories(s["path"])
|
||||
out_sources.append({"source": s["name"], "type": s["type"],
|
||||
"editable": src_module.is_editable_source(s), "inventories": inventories})
|
||||
for path in src_module.discover_playbooks(s["path"]):
|
||||
meta = src_module.discover_playbook_meta(src_module.read_playbook(s["path"], path) or "")
|
||||
if category and meta["category"].lower() != category.strip().lower():
|
||||
continue
|
||||
rows.append({"source": s["name"], "playbook": path, **meta})
|
||||
return {"sources": out_sources,
|
||||
**bounded(rows, limit, key="playbooks", more="Filter by `source` or `category`.")}
|
||||
|
||||
|
||||
async def get_playbook(source: str, playbook: str) -> dict:
|
||||
"""One playbook: its description, the variables it takes, and its content.
|
||||
|
||||
`variables` are what run_playbook's extra_vars can set: `required` ones
|
||||
(a vars_prompt with no default) must be given; `secret` ones are passed
|
||||
to the run but never stored, and their defaults are shown as "<secret>".
|
||||
`content` is the YAML itself, cut at 20000 characters.
|
||||
"""
|
||||
s = _source(source)
|
||||
content = _playbook(s, playbook)
|
||||
variables = src_module.discover_playbook_variables(content)
|
||||
_, filled = steward_defaults(s, variables, {})
|
||||
variables = [{**v, "default": filled[v["name"]], "filled_by_steward": True}
|
||||
if v["name"] in filled else v for v in variables]
|
||||
out = {
|
||||
"source": s["name"],
|
||||
"playbook": playbook.strip(),
|
||||
**src_module.discover_playbook_meta(content),
|
||||
"variables": public_variables(variables),
|
||||
"content": content[:CONTENT_CHARS],
|
||||
}
|
||||
if len(content) > CONTENT_CHARS:
|
||||
out["content_truncated"] = f"{len(content) - CONTENT_CHARS} more characters not shown."
|
||||
return out
|
||||
|
||||
|
||||
async def list_inventory() -> dict:
|
||||
"""Steward's Ansible inventory: targets, groups, and how to scope a run to them.
|
||||
|
||||
Each target names the Steward host it is linked to (prune_docker needs
|
||||
that link), its groups and the names of its Ansible variables — not their
|
||||
values, which can hold credentials. `scopes` lists what run_playbook's
|
||||
`scope` accepts.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
targets = list((await db.execute(
|
||||
select(AnsibleTarget).options(selectinload(AnsibleTarget.groups))
|
||||
.order_by(AnsibleTarget.name))).scalars())
|
||||
groups = list((await db.execute(
|
||||
select(AnsibleGroup).options(selectinload(AnsibleGroup.targets))
|
||||
.order_by(AnsibleGroup.name))).scalars())
|
||||
host_names = dict((await db.execute(select(Host.id, Host.name))).all())
|
||||
return {
|
||||
"targets": [
|
||||
{"target_id": t.id, "name": t.name, "address": t.address, "host_id": t.host_id,
|
||||
"host_name": host_names.get(t.host_id), "groups": sorted(g.name for g in t.groups),
|
||||
"var_names": sorted(t.ansible_vars or {})}
|
||||
for t in targets
|
||||
],
|
||||
"groups": [
|
||||
{"group_id": g.id, "name": g.name, "targets": sorted(t.name for t in g.targets),
|
||||
"var_names": sorted(g.ansible_vars or {})}
|
||||
for g in groups
|
||||
],
|
||||
"repo_inventories": [{"source": s["name"], "files": inv} for s in _sources()
|
||||
if (inv := src_module.discover_inventories(s["path"]))],
|
||||
"scopes": {
|
||||
"all": "every target above",
|
||||
"target:<name or id>": "one target",
|
||||
"group:<name or id>": "every target in a group",
|
||||
"inventory:<file>": "an inventory file in the playbook's own source",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ── runs ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
def recap_totals(results: dict | None) -> dict | None:
|
||||
hosts = (results or {}).get("hosts") or {}
|
||||
if not hosts:
|
||||
return None
|
||||
return {"hosts": len(hosts),
|
||||
**{k: sum(int((h or {}).get(k) or 0) for h in hosts.values()) for k in RECAP_KEYS}}
|
||||
|
||||
|
||||
def run_row(r: AnsibleRun, users: dict, labels: dict) -> dict:
|
||||
"""`labels` is scope_labels() for the runs' scopes."""
|
||||
end = r.finished_at
|
||||
return {
|
||||
"run_id": r.id,
|
||||
"source": r.source_name,
|
||||
"playbook": r.playbook_path,
|
||||
"scope": r.inventory_scope,
|
||||
"scope_label": labels.get(r.inventory_scope, r.inventory_scope),
|
||||
"status": r.status.value,
|
||||
"check": bool((r.params or {}).get("check")),
|
||||
"triggered_by": users.get(r.triggered_by, "(deleted user)") if r.triggered_by else "system",
|
||||
"started_at": iso(r.started_at),
|
||||
"age_s": age_s(r.started_at),
|
||||
"finished_at": iso(end),
|
||||
"duration_s": int((end - r.started_at).total_seconds()) if end and r.started_at else None,
|
||||
"recap": recap_totals(r.results),
|
||||
}
|
||||
|
||||
|
||||
async def _users(db, ids) -> dict[str, str]:
|
||||
ids = {i for i in ids if i}
|
||||
if not ids:
|
||||
return {}
|
||||
return dict((await db.execute(select(User.id, User.username).where(User.id.in_(ids)))).all())
|
||||
|
||||
|
||||
async def list_ansible_runs(status: str = "", playbook: str = "", range: str = "7d",
|
||||
limit: int = 50) -> dict:
|
||||
"""Ansible runs, newest first: manual, scheduled, alert-triggered and MCP-started.
|
||||
|
||||
status: queued, running, success, failed, interrupted or cancelled; empty for all.
|
||||
playbook: a substring of the playbook path; empty for all.
|
||||
range: "1h", "6h", "24h", "7d" (default) or "30d".
|
||||
limit: default 50, max 500.
|
||||
|
||||
`check: true` is a dry run that changed nothing. `recap` sums the play
|
||||
recap over hosts; get_ansible_run shows per-host results and the log.
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
limit = clamp_limit(limit, 50, 500)
|
||||
stmt = (select(AnsibleRun).where(AnsibleRun.started_at >= since)
|
||||
.order_by(AnsibleRun.started_at.desc()).limit(limit + 1))
|
||||
if status:
|
||||
try:
|
||||
stmt = stmt.where(AnsibleRun.status == AnsibleRunStatus(status.strip()))
|
||||
except ValueError:
|
||||
raise ValueError(f"status {status!r} is not one of "
|
||||
f"{', '.join(s.value for s in AnsibleRunStatus)}.") from None
|
||||
if playbook:
|
||||
esc = playbook.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
stmt = stmt.where(AnsibleRun.playbook_path.ilike(f"%{esc}%", escape="\\"))
|
||||
async with db_session() as db:
|
||||
runs = list((await db.execute(stmt)).scalars())
|
||||
users = await _users(db, [r.triggered_by for r in runs])
|
||||
labels = await scope_labels(db, [r.inventory_scope for r in runs])
|
||||
out = {"range": key, **bounded([run_row(r, users, labels) for r in runs], limit, key="runs")}
|
||||
if out["truncated"]:
|
||||
out["total"] = None
|
||||
out["more"] = (f"More than {limit} runs in range. Pass a larger `limit` (max 500), "
|
||||
"a shorter `range`, or filter by `status` or `playbook`.")
|
||||
return out
|
||||
|
||||
|
||||
def _cut(line: str) -> str:
|
||||
return line if len(line) <= LINE_CHARS else line[:LINE_CHARS] + f" [… {len(line) - LINE_CHARS} more]"
|
||||
|
||||
|
||||
def read_log(run: AnsibleRun, tail: int, grep: str, max_matches: int = 100) -> dict:
|
||||
"""The last `tail` lines of a run's log and the lines containing `grep`."""
|
||||
path = executor.artifact_path(run.id)
|
||||
if os.path.exists(path):
|
||||
with open(path, encoding="utf-8", errors="replace") as f:
|
||||
lines = f.read().splitlines()
|
||||
else:
|
||||
lines = (run.output or "").splitlines()
|
||||
out: dict = {"log_lines": len(lines), "tail": [_cut(x) for x in deque(lines, maxlen=tail)]}
|
||||
if grep:
|
||||
needle = grep.lower()
|
||||
hits = [{"line_no": i + 1, "line": _cut(x)} for i, x in enumerate(lines) if needle in x.lower()]
|
||||
out["matches"] = hits[:max_matches]
|
||||
out["matches_total"] = len(hits)
|
||||
return out
|
||||
|
||||
|
||||
async def _load_run(db, run_id: str) -> AnsibleRun:
|
||||
run = await db.get(AnsibleRun, (run_id or "").strip())
|
||||
if run is None:
|
||||
raise ValueError(f"no Ansible run with id {run_id!r}; list_ansible_runs lists them.")
|
||||
return run
|
||||
|
||||
|
||||
async def _run_detail(run_id: str, tail: int, grep: str) -> dict:
|
||||
async with db_session() as db:
|
||||
run = await _load_run(db, run_id)
|
||||
users = await _users(db, [run.triggered_by])
|
||||
labels = await scope_labels(db, [run.inventory_scope])
|
||||
params = dict(run.params or {})
|
||||
return {
|
||||
**run_row(run, users, labels),
|
||||
"limit": params.get("limit"),
|
||||
"tags": params.get("tags"),
|
||||
"extra_vars": params.get("extra_vars_map") or {},
|
||||
"hosts": (run.results or {}).get("hosts") or {},
|
||||
"failures": ((run.results or {}).get("failures") or [])[:20],
|
||||
**read_log(run, tail, grep),
|
||||
}
|
||||
|
||||
|
||||
async def get_ansible_run(run_id: str, tail: int = 200, grep: str = "") -> dict:
|
||||
"""One Ansible run: its parameters, per-host recap, failures and log.
|
||||
|
||||
tail: how many of the log's last lines to return; default 200, max 2000.
|
||||
grep: also return every log line containing this text (case-insensitive),
|
||||
with its line number, up to 100.
|
||||
|
||||
`failures` are the task failures Steward parsed out of the log; start
|
||||
there when status is failed. Secret variables were never stored, so they
|
||||
do not appear in `extra_vars`.
|
||||
"""
|
||||
return await _run_detail(run_id, clamp_limit(tail, 200, 2000), (grep or "").strip())
|
||||
|
||||
|
||||
async def wait_for_run(run_id: str, timeout_s: int = 60) -> dict:
|
||||
"""Wait for a run to finish, then return it as get_ansible_run does (last 50 lines).
|
||||
|
||||
timeout_s: how long to wait; default 60, max 120. If the run is still
|
||||
going when it runs out, `finished` is false: call again to keep waiting.
|
||||
"""
|
||||
deadline = asyncio.get_running_loop().time() + max(1, min(int(timeout_s or 60), WAIT_MAX_S))
|
||||
while True:
|
||||
async with db_session() as db:
|
||||
status = (await _load_run(db, run_id)).status
|
||||
if status not in ACTIVE or asyncio.get_running_loop().time() >= deadline:
|
||||
break
|
||||
await asyncio.sleep(POLL_S)
|
||||
out = await _run_detail(run_id, 50, "")
|
||||
out["finished"] = status not in ACTIVE
|
||||
return out
|
||||
|
||||
|
||||
async def run_playbook(source: str, playbook: str, scope: str = "all",
|
||||
extra_vars: dict | None = None, limit: str = "", tags: str = "",
|
||||
check: bool = True, confirmed: bool = False) -> dict:
|
||||
"""Start an Ansible playbook run. A dry run (check mode) unless check=false.
|
||||
|
||||
source, playbook: as list_playbooks shows them.
|
||||
scope: which hosts — "all" (default), "target:<name or id>",
|
||||
"group:<name or id>", or "inventory:<file>" for an inventory file in the
|
||||
same source. list_inventory shows the choices.
|
||||
extra_vars: {"name": value}. Variables the playbook marks secret are
|
||||
passed to the run but never stored or returned. Steward's built-in
|
||||
playbooks get `steward_url` (Steward's own address) filled in when it is
|
||||
left out; the reply's `filled_by_steward` says so.
|
||||
limit, tags: Ansible's --limit and --tags; empty for none.
|
||||
check: true (default) runs with --check --diff, which reports what would
|
||||
change and changes nothing. Pass false to apply — after a check run
|
||||
looked right.
|
||||
confirmed: a playbook with `confirm: true` (destructive, by its author's
|
||||
flag) applies only when this is also true.
|
||||
|
||||
Returns at once with `run_id`; wait_for_run follows it to the end. Recorded
|
||||
in the audit log as "<you> via MCP".
|
||||
"""
|
||||
s = _source(source)
|
||||
content = _playbook(s, playbook)
|
||||
meta = src_module.discover_playbook_meta(content)
|
||||
if not check and meta["confirm"] and not confirmed:
|
||||
raise ValueError(f"{playbook} is flagged destructive by its author (steward:confirm). "
|
||||
"Run it with check=true first; to apply, pass check=false and "
|
||||
"confirmed=true. Nothing was run.")
|
||||
variables = src_module.discover_playbook_variables(content)
|
||||
extra_vars, filled = steward_defaults(s, variables, extra_vars)
|
||||
plain, secret = split_vars(variables, extra_vars)
|
||||
async with db_session() as db:
|
||||
stored_scope, scope_desc = await _scope(db, s, scope)
|
||||
params = build_params(plain, limit, tags, check)
|
||||
caller = current_caller()
|
||||
run, _src, err = await trigger_run(
|
||||
mcp_app(), source_name=s["name"], playbook_path=playbook.strip(),
|
||||
inventory_scope=stored_scope, params=params or None,
|
||||
secret_vars=secret or None, triggered_by=caller.user_id)
|
||||
if err:
|
||||
raise ValueError(f"the run could not start: {err}.")
|
||||
await audit_mcp("ansible.run.started", "ansible_run", run.id, {
|
||||
"source": s["name"], "playbook": playbook.strip(), "scope": stored_scope,
|
||||
"scope_label": scope_desc, "check": check, "limit": params.get("limit"), "tags": params.get("tags"),
|
||||
"extra_vars": sorted(plain), "secret_vars": sorted(secret)})
|
||||
return {
|
||||
"run_id": run.id,
|
||||
"check": check,
|
||||
"scope": scope_desc,
|
||||
"status": run.status.value,
|
||||
"secret_vars_passed": sorted(secret),
|
||||
**({"filled_by_steward": filled} if filled else {}),
|
||||
"summary": (f"{'Check-mode (dry) run' if check else 'Run'} of {playbook.strip()} "
|
||||
f"on {scope_desc} started."
|
||||
+ (" It changes nothing; pass check=false to apply." if check else "")),
|
||||
"next": f'wait_for_run(run_id="{run.id}")',
|
||||
}
|
||||
|
||||
|
||||
async def cancel_ansible_run(run_id: str) -> dict:
|
||||
"""Stop a queued or running Ansible run. Recorded in the audit log."""
|
||||
async with db_session() as db:
|
||||
run = await _load_run(db, run_id)
|
||||
if run.status not in ACTIVE:
|
||||
raise ValueError(f"run {run.id} already ended ({run.status.value}); nothing to cancel.")
|
||||
executor.cancel_run(run.id)
|
||||
await audit_mcp("ansible.run.cancelled", "ansible_run", run.id, {"playbook": run.playbook_path})
|
||||
return {"run_id": run.id, "cancelling": True,
|
||||
"next": f'wait_for_run(run_id="{run.id}", timeout_s=30)'}
|
||||
|
||||
|
||||
# ── schedules ────────────────────────────────────────────────────────────────
|
||||
|
||||
def schedule_row(sc: AnsibleSchedule, last: AnsibleRun | None, labels: dict) -> dict:
|
||||
params = sc.params or {}
|
||||
nxt = sc.last_run_at + timedelta(seconds=sc.interval_seconds) if sc.last_run_at else None
|
||||
return {
|
||||
"schedule_id": sc.id,
|
||||
"name": sc.name,
|
||||
"source": sc.source_name,
|
||||
"playbook": sc.playbook_path,
|
||||
"scope": sc.inventory_scope,
|
||||
"scope_label": labels.get(sc.inventory_scope, sc.inventory_scope),
|
||||
"interval_s": sc.interval_seconds,
|
||||
"enabled": sc.enabled,
|
||||
"check": bool(params.get("check")),
|
||||
"extra_vars": params.get("extra_vars_map") or {},
|
||||
"limit": params.get("limit"),
|
||||
"tags": params.get("tags"),
|
||||
"last_run_at": iso(sc.last_run_at),
|
||||
"last_run_id": sc.last_run_id,
|
||||
"last_run_status": last.status.value if last else None,
|
||||
"last_error": sc.last_error,
|
||||
"next_run_at": iso(nxt) if sc.enabled else None,
|
||||
}
|
||||
|
||||
|
||||
async def _schedules(db) -> list[AnsibleSchedule]:
|
||||
return list((await db.execute(select(AnsibleSchedule).order_by(AnsibleSchedule.name))).scalars())
|
||||
|
||||
|
||||
async def list_schedules() -> dict:
|
||||
"""Recurring playbook runs: what runs, how often, and how its last run went.
|
||||
|
||||
`check: true` schedules are dry runs that never change anything.
|
||||
`last_error` is set when a firing could not start at all (a source or
|
||||
playbook that went away); a run that started and failed shows in
|
||||
`last_run_status`, and get_ansible_run(last_run_id) has its log.
|
||||
"""
|
||||
async with db_session() as db:
|
||||
scheds = await _schedules(db)
|
||||
ids = [s.last_run_id for s in scheds if s.last_run_id]
|
||||
runs = {r.id: r for r in (await db.execute(
|
||||
select(AnsibleRun).where(AnsibleRun.id.in_(ids)))).scalars()} if ids else {}
|
||||
labels = await scope_labels(db, [s.inventory_scope for s in scheds])
|
||||
rows = [schedule_row(s, runs.get(s.last_run_id), labels) for s in scheds]
|
||||
return {"schedules": rows, "total": len(rows)}
|
||||
|
||||
|
||||
async def _schedule_fields(db, source: str, playbook: str, scope: str, extra_vars: dict | None,
|
||||
limit: str, tags: str, check: bool) -> dict:
|
||||
s = _source(source)
|
||||
content = _playbook(s, playbook)
|
||||
variables = src_module.discover_playbook_variables(content)
|
||||
plain, secret = split_vars(variables, steward_defaults(s, variables, extra_vars)[0])
|
||||
if secret:
|
||||
raise ValueError(f"{', '.join(sorted(secret))} {'is' if len(secret) == 1 else 'are'} secret "
|
||||
"and a schedule would have to store it. Put it in the target's or group's "
|
||||
"Ansible variables instead. Nothing was saved.")
|
||||
stored_scope, _desc = await _scope(db, s, scope)
|
||||
return {"source_name": s["name"], "playbook_path": playbook.strip(), "inventory_scope": stored_scope,
|
||||
"params": build_params(plain, limit, tags, check) or None}
|
||||
|
||||
|
||||
def _interval(interval_s: int) -> int:
|
||||
if not isinstance(interval_s, int) or interval_s <= 0:
|
||||
raise ValueError("interval_s must be a positive number of seconds, e.g. 86400 for daily.")
|
||||
return interval_s
|
||||
|
||||
|
||||
async def create_schedule(name: str, source: str, playbook: str, interval_s: int,
|
||||
scope: str = "all", extra_vars: dict | None = None, limit: str = "",
|
||||
tags: str = "", check: bool = True, enabled: bool = True) -> dict:
|
||||
"""Run a playbook on a recurring interval. Each firing is a "system" run.
|
||||
|
||||
interval_s: seconds between runs, e.g. 3600 hourly, 86400 daily.
|
||||
scope, extra_vars, limit, tags: as run_playbook. Secret variables cannot
|
||||
be scheduled (they would have to be stored); set them as inventory vars.
|
||||
check: true (default) schedules dry runs. Pass false only for a playbook
|
||||
already proven by a check run.
|
||||
|
||||
Recorded in the audit log as "<you> via MCP".
|
||||
"""
|
||||
if not (name or "").strip():
|
||||
raise ValueError("a schedule needs a name.")
|
||||
async with db_session() as db, db.begin():
|
||||
fields = await _schedule_fields(db, source, playbook, scope, extra_vars, limit, tags, check)
|
||||
sc = AnsibleSchedule(name=name.strip()[:128], interval_seconds=_interval(interval_s),
|
||||
enabled=enabled, **fields)
|
||||
db.add(sc)
|
||||
labels = await scope_labels(db, [sc.inventory_scope])
|
||||
await audit_mcp("ansible.schedule.created", "ansible_schedule", sc.id,
|
||||
{"name": sc.name, "playbook": sc.playbook_path, "interval_s": sc.interval_seconds,
|
||||
"check": check})
|
||||
return schedule_row(sc, None, labels)
|
||||
|
||||
|
||||
async def update_schedule(schedule: str, name: str | None = None, source: str | None = None,
|
||||
playbook: str | None = None, interval_s: int | None = None,
|
||||
scope: str | None = None, extra_vars: dict | None = None,
|
||||
limit: str | None = None, tags: str | None = None,
|
||||
check: bool | None = None, enabled: bool | None = None) -> dict:
|
||||
"""Change a schedule. Only the arguments you pass change; the rest stay.
|
||||
|
||||
schedule: its name or id (list_schedules). Pass enabled=false to pause it.
|
||||
extra_vars, when passed, replaces the whole set. Recorded in the audit log.
|
||||
"""
|
||||
async with db_session() as db, db.begin():
|
||||
sc = pick(schedule, await _schedules(db), kind="schedule", list_tool="list_schedules")
|
||||
params = dict(sc.params or {})
|
||||
cur_scope = sc.inventory_scope
|
||||
if cur_scope.startswith("repo:"):
|
||||
cur_scope = "inventory:" + cur_scope.rsplit(":", 1)[-1]
|
||||
fields = await _schedule_fields(
|
||||
db, source if source is not None else sc.source_name,
|
||||
playbook if playbook is not None else sc.playbook_path,
|
||||
scope if scope is not None else cur_scope,
|
||||
extra_vars if extra_vars is not None else params.get("extra_vars_map"),
|
||||
limit if limit is not None else params.get("limit", ""),
|
||||
tags if tags is not None else params.get("tags", ""),
|
||||
check if check is not None else bool(params.get("check")))
|
||||
for k, v in fields.items():
|
||||
setattr(sc, k, v)
|
||||
if name is not None and name.strip():
|
||||
sc.name = name.strip()[:128]
|
||||
if interval_s is not None:
|
||||
sc.interval_seconds = _interval(interval_s)
|
||||
if enabled is not None:
|
||||
sc.enabled = enabled
|
||||
row = schedule_row(sc, None, await scope_labels(db, [sc.inventory_scope]))
|
||||
changed = sorted(k for k, v in {"name": name, "source": source, "playbook": playbook,
|
||||
"interval_s": interval_s, "scope": scope, "extra_vars": extra_vars,
|
||||
"limit": limit, "tags": tags, "check": check,
|
||||
"enabled": enabled}.items() if v is not None)
|
||||
await audit_mcp("ansible.schedule.updated", "ansible_schedule", row["schedule_id"],
|
||||
{"name": row["name"], "changed": changed})
|
||||
return row
|
||||
|
||||
|
||||
async def delete_schedule(schedule: str) -> dict:
|
||||
"""Delete a schedule (its past runs stay). Recorded in the audit log.
|
||||
|
||||
schedule: its name or id. To pause instead, update_schedule(enabled=false).
|
||||
"""
|
||||
async with db_session() as db, db.begin():
|
||||
sc = pick(schedule, await _schedules(db), kind="schedule", list_tool="list_schedules")
|
||||
await db.delete(sc)
|
||||
await audit_mcp("ansible.schedule.deleted", "ansible_schedule", sc.id,
|
||||
{"name": sc.name, "playbook": sc.playbook_path})
|
||||
return {"deleted": sc.name, "schedule_id": sc.id}
|
||||
|
||||
|
||||
# ── alert actions ────────────────────────────────────────────────────────────
|
||||
|
||||
async def set_alert_action(rule: str, source: str = "", playbook: str = "", inventory: str = "",
|
||||
extra_vars: dict | None = None, limit: str = "", tags: str = "",
|
||||
check: bool = True) -> dict:
|
||||
"""Set, replace or clear the playbook an alert rule runs when it starts firing.
|
||||
|
||||
rule: the alert rule's name or id (list_alert_rules).
|
||||
playbook: empty clears the rule's action. Otherwise source, playbook and
|
||||
inventory (an inventory file in that source, as list_playbooks shows)
|
||||
are all required.
|
||||
extra_vars: {"name": value}, stored on the rule. The run also receives the
|
||||
alert's own context variables. Secret variables cannot be stored here.
|
||||
check: true (default) makes the action a dry run each time it fires;
|
||||
pass false for a remediation already proven by a check run.
|
||||
|
||||
Admin only, as on the alert rule form. Recorded in the audit log.
|
||||
"""
|
||||
async with db_session() as db, db.begin():
|
||||
r = pick(rule, (await db.execute(select(AlertRule))).scalars(),
|
||||
kind="alert rule", list_tool="list_alert_rules")
|
||||
action = None
|
||||
if (playbook or "").strip():
|
||||
s = _source(source)
|
||||
content = _playbook(s, playbook)
|
||||
inv = (inventory or "").strip()
|
||||
found = src_module.discover_inventories(s["path"])
|
||||
if inv not in found:
|
||||
raise ValueError(f"source {s['name']!r} has no inventory file {inv!r}"
|
||||
+ (f"; it has {', '.join(found)}." if found else "; it has none.")
|
||||
+ " Nothing was saved.")
|
||||
variables = src_module.discover_playbook_variables(content)
|
||||
plain, secret = split_vars(variables, steward_defaults(s, variables, extra_vars)[0])
|
||||
if secret:
|
||||
raise ValueError(f"{', '.join(sorted(secret))} is secret and an alert action would "
|
||||
"have to store it. Nothing was saved.")
|
||||
action = {"source": s["name"], "playbook": playbook.strip(), "inventory": inv}
|
||||
if plain:
|
||||
action["extra_vars"] = [f"{k}={v}" for k, v in plain.items()]
|
||||
if (limit or "").strip():
|
||||
action["limit"] = limit.strip()
|
||||
if (tags or "").strip():
|
||||
action["tags"] = tags.strip()
|
||||
if check:
|
||||
action["check"] = True
|
||||
r.ansible_action = action
|
||||
name, rid = r.name, r.id
|
||||
await audit_mcp("alert_rule.action_set" if action else "alert_rule.action_cleared",
|
||||
"alert_rule", name,
|
||||
{"rule_id": rid, **({k: v for k, v in action.items() if k != "extra_vars"}
|
||||
if action else {})})
|
||||
return {"rule_id": rid, "rule_name": name, "action": action,
|
||||
"summary": (f"When {name} starts firing it runs {action['playbook']}"
|
||||
f"{' in check mode' if action.get('check') else ''}."
|
||||
if action else f"{name} no longer runs a playbook when it fires.")}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
for fn in (list_playbooks, get_playbook, list_inventory, list_ansible_runs,
|
||||
get_ansible_run, list_schedules, wait_for_run):
|
||||
register_tool(fn, role=UserRole.viewer, read_only=True)
|
||||
for fn in (run_playbook, cancel_ansible_run, create_schedule, update_schedule, delete_schedule):
|
||||
register_tool(fn, role=UserRole.operator, read_only=False)
|
||||
register_tool(set_alert_action, role=UserRole.admin, read_only=False)
|
||||
@@ -1,275 +0,0 @@
|
||||
"""Host tools: the fleet, one host in depth, and a host's agent metrics."""
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.core import metric_series
|
||||
from steward.hosts import service as host_svc
|
||||
from steward.mcp._context import mcp_app
|
||||
from steward.models.ansible_inventory import AnsibleTarget
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from ._shape import (
|
||||
age_s, bounded, clamp_limit, db_session, iso, lookback, pick, rnd, summarize,
|
||||
)
|
||||
|
||||
AGENT_SOURCE = "host_agent"
|
||||
|
||||
|
||||
async def _all_hosts(db) -> list[Host]:
|
||||
return list((await db.execute(select(Host).order_by(Host.name))).scalars())
|
||||
|
||||
|
||||
def _agent_glance(a: dict | None) -> dict | None:
|
||||
if not a:
|
||||
return None
|
||||
return {
|
||||
"reporting": a.get("fresh", False),
|
||||
"last_seen_at": iso(a.get("last_at")),
|
||||
"last_seen_age_s": age_s(a.get("last_at")),
|
||||
"cpu_pct": rnd(a.get("cpu_pct")),
|
||||
"mem_used_pct": rnd(a.get("mem_used_pct")),
|
||||
"disk_used_pct_worst": rnd(a.get("disk_used_pct_worst")),
|
||||
"disk_root_used_pct": rnd(a.get("disk_root")),
|
||||
}
|
||||
|
||||
|
||||
_PART_GROUPS = ( # (prefix, group, the field that names one part)
|
||||
("net:", "interfaces", "interface"),
|
||||
("diskio:", "disk_io", "device"),
|
||||
("temp:", "temperatures", "sensor"),
|
||||
)
|
||||
# Sort each group by what makes a part worth a look first.
|
||||
_PART_ORDER = {
|
||||
"mounts": lambda r: -(r.get("disk_used_pct") or 0),
|
||||
"interfaces": lambda r: -((r.get("net_rx_bps") or 0) + (r.get("net_tx_bps") or 0)),
|
||||
"disk_io": lambda r: -((r.get("disk_read_bps") or 0) + (r.get("disk_write_bps") or 0)),
|
||||
"temperatures": lambda r: -(r.get("temp_c") or 0),
|
||||
"other": lambda r: r["part"],
|
||||
}
|
||||
|
||||
|
||||
def agent_parts(parts: dict[str, dict], host_name: str, limit: int) -> dict:
|
||||
"""A host's sub-resources grouped by kind, each group bounded on its own.
|
||||
|
||||
`parts` maps the part name (the resource after "<host>:") to its latest
|
||||
metrics. Per-core CPU is summarised, not listed: the host's cpu_pct
|
||||
already says how busy it is, and 30+ cores would crowd out the rest.
|
||||
"""
|
||||
groups: dict[str, list] = {"mounts": [], "interfaces": [], "disk_io": [],
|
||||
"temperatures": [], "other": []}
|
||||
cores = []
|
||||
for name, metrics in parts.items():
|
||||
if name.startswith("core") and name[4:].isdigit():
|
||||
if metrics.get("cpu_pct") is not None:
|
||||
cores.append((int(name[4:]), metrics["cpu_pct"]))
|
||||
continue
|
||||
for prefix, group, field in _PART_GROUPS:
|
||||
if name.startswith(prefix):
|
||||
groups[group].append({field: name[len(prefix):], "part": name, **metrics})
|
||||
break
|
||||
else:
|
||||
if any(m.startswith("disk_") for m in metrics):
|
||||
groups["mounts"].append({"mount": name, "part": name, **metrics})
|
||||
else:
|
||||
groups["other"].append({"part": name, **metrics})
|
||||
more = (f"Pass a larger `parts_limit` (max 200), or read one part with query_metrics("
|
||||
f'source_module="host_agent", resource="{host_name}:<part>").')
|
||||
out: dict = {}
|
||||
for group, rows in groups.items():
|
||||
if rows or group != "other":
|
||||
rows.sort(key=_PART_ORDER[group])
|
||||
out[group] = bounded(rows, limit, key="items", more=more)
|
||||
if cores:
|
||||
pcts = [pct for _, pct in cores]
|
||||
busiest = sorted(cores, key=lambda c: (-c[1], c[0]))[:3]
|
||||
out["cpu_cores"] = {
|
||||
"count": len(cores), "min_pct": rnd(min(pcts)),
|
||||
"avg_pct": rnd(sum(pcts) / len(pcts)), "max_pct": rnd(max(pcts)),
|
||||
"busiest": [{"core": i, "cpu_pct": rnd(pct)} for i, pct in busiest],
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
async def list_hosts(limit: int = 100) -> dict:
|
||||
"""Every registered host with its health at a glance.
|
||||
|
||||
For each host: `status` rolled up from its linked monitors ("down" if any
|
||||
monitor's latest check failed, "up", "pending" when none has run yet,
|
||||
"unmonitored" when it has no monitors), the best latency, uptime % over
|
||||
24h/7d/30d, and — when a host agent reports for it — CPU, memory and disk
|
||||
utilisation and whether the agent is still reporting (`agent.reporting`;
|
||||
false means its last sample is older than the stale window).
|
||||
|
||||
Use get_host for one host's monitors, alerts and Ansible link, and
|
||||
get_host_metrics for a metric's history. limit: default 100, max 500.
|
||||
"""
|
||||
limit = clamp_limit(limit, 100, 500)
|
||||
async with db_session() as db:
|
||||
hosts = await _all_hosts(db)
|
||||
status = await host_svc.host_status(db, [h.id for h in hosts])
|
||||
uptime = await host_svc.compute_uptime(db)
|
||||
agent = await host_svc.agent_overview_by_host(
|
||||
db, [h.name for h in hosts], host_svc.agent_stale_after_seconds(mcp_app().config))
|
||||
rows = []
|
||||
for h in hosts:
|
||||
st = status.get(h.id)
|
||||
rows.append({
|
||||
"host_id": h.id,
|
||||
"host_name": h.name,
|
||||
"address": h.address,
|
||||
"status": st["state"] if st else "unmonitored",
|
||||
"monitor_count": st["count"] if st else 0,
|
||||
"best_latency_ms": rnd(st["latency_ms"]) if st else None,
|
||||
"uptime_pct": uptime.get(h.id) or {"24h": None, "7d": None, "30d": None},
|
||||
"agent": _agent_glance(agent.get(h.name)),
|
||||
})
|
||||
return bounded(rows, limit, key="hosts")
|
||||
|
||||
|
||||
async def get_host(host: str, parts_limit: int = 20) -> dict:
|
||||
"""One host in depth: its monitors, agent snapshot, active alerts and Ansible link.
|
||||
|
||||
host: the host's name (case-insensitive) or id, as list_hosts shows.
|
||||
parts_limit: how many of each kind of sub-resource to list (default 20,
|
||||
max 200).
|
||||
|
||||
Returns the host's monitors (latest result, error, uptime), the agent's
|
||||
latest values, alerts currently pending, firing or acknowledged on this
|
||||
host's resources, and the Ansible inventory target linked to it with its
|
||||
groups (`ansible_target`: null when unlinked).
|
||||
|
||||
The agent's values are `agent.metrics` (host-level) and its sub-resources
|
||||
grouped by kind: `mounts` (fullest first), `interfaces` and `disk_io`
|
||||
(busiest first), `temperatures` (hottest first), each with `items`,
|
||||
`total` and `truncated`; `cpu_cores` summarises per-core CPU (count,
|
||||
min/avg/max, the three busiest). One part's history is query_metrics with
|
||||
source_module="host_agent" and resource="<host>:<part>", where `part` is
|
||||
the item's `part`.
|
||||
"""
|
||||
parts_limit = clamp_limit(parts_limit, 20, 200)
|
||||
async with db_session() as db:
|
||||
h = pick(host, await _all_hosts(db), kind="host", list_tool="list_hosts")
|
||||
monitors = await host_svc.host_monitors(db, h.id)
|
||||
uptime = (await host_svc.compute_uptime(db)).get(h.id)
|
||||
status = (await host_svc.host_status(db, [h.id])).get(h.id)
|
||||
glance = (await host_svc.agent_overview_by_host(
|
||||
db, [h.name], host_svc.agent_stale_after_seconds(mcp_app().config))).get(h.name)
|
||||
samples = await metric_series.latest_values(db, AGENT_SOURCE, h.name, include_children=True)
|
||||
active = await alert_svc.active_alerts(db)
|
||||
target = (await db.execute(
|
||||
select(AnsibleTarget).where(AnsibleTarget.host_id == h.id)
|
||||
.options(selectinload(AnsibleTarget.groups))
|
||||
)).scalar_one_or_none()
|
||||
|
||||
agent = None
|
||||
if samples:
|
||||
host_level: dict = {}
|
||||
parts: dict[str, dict] = {}
|
||||
for s in samples:
|
||||
if s.resource_name == h.name:
|
||||
host_level[s.metric_name] = rnd(s.value)
|
||||
else:
|
||||
parts.setdefault(s.resource_name[len(h.name) + 1:], {})[s.metric_name] = rnd(s.value)
|
||||
agent = {
|
||||
**(_agent_glance(glance) or {}),
|
||||
"metrics": host_level,
|
||||
**agent_parts(parts, h.name, parts_limit),
|
||||
}
|
||||
|
||||
prefix = h.name + ":"
|
||||
alerts = [
|
||||
{
|
||||
"rule_id": rule.id, "rule_name": rule.name, "state": state.state.value,
|
||||
"resource": rule.resource_name, "metric": rule.metric_name,
|
||||
"condition": f"{rule.metric_name} {rule.operator.value} {rule.threshold:g}",
|
||||
"since": iso(state.last_transition_at), "age_s": age_s(state.last_transition_at),
|
||||
}
|
||||
for state, rule in active
|
||||
if rule.resource_name == h.name or rule.resource_name.startswith(prefix)
|
||||
]
|
||||
|
||||
return {
|
||||
"host_id": h.id,
|
||||
"host_name": h.name,
|
||||
"address": h.address,
|
||||
"status": status["state"] if status else "unmonitored",
|
||||
"uptime_pct": uptime or {"24h": None, "7d": None, "30d": None},
|
||||
"monitors": [
|
||||
{
|
||||
"monitor_id": d["monitor"].id,
|
||||
"monitor_name": d["monitor"].name,
|
||||
"type": d["monitor"].type,
|
||||
"target": d["monitor"].target,
|
||||
"enabled": d["monitor"].enabled,
|
||||
"state": ("pending" if d["latest"] is None
|
||||
else "up" if d["latest"].is_up else "down"),
|
||||
"last_checked_at": iso(d["latest"].checked_at) if d["latest"] else None,
|
||||
"response_ms": rnd(d["latest"].response_ms) if d["latest"] else None,
|
||||
"error": d["latest"].error_msg if d["latest"] and not d["latest"].is_up else None,
|
||||
"uptime_24h_pct": d["uptime_pct"],
|
||||
}
|
||||
for d in monitors
|
||||
],
|
||||
"agent": agent,
|
||||
"active_alerts": alerts,
|
||||
"ansible_target": {
|
||||
"target_id": target.id,
|
||||
"target_name": target.name,
|
||||
"address": target.address,
|
||||
"groups": sorted(g.name for g in target.groups),
|
||||
} if target else None,
|
||||
}
|
||||
|
||||
|
||||
async def get_host_metrics(host: str, metric: str = "", range: str = "24h") -> dict:
|
||||
"""A host agent metric over time: summary statistics plus a ≤60-point series.
|
||||
|
||||
host: name or id, as list_hosts shows.
|
||||
metric: e.g. "cpu_pct", "mem_used_pct", "disk_used_pct_worst", "load_1m",
|
||||
"net_rx_bps", "temp_c_max". Empty lists the metrics this host reports,
|
||||
with their latest values. For a mount or interface, use query_metrics
|
||||
with source_module="host_agent" and resource="<host>:<part>" (get_host
|
||||
lists the parts).
|
||||
range: "1h", "6h", "24h" (default), "7d" or "30d".
|
||||
|
||||
The summary carries min, avg, max, p95 and last over the range; each series
|
||||
point is [time, value].
|
||||
"""
|
||||
since, key = lookback(range)
|
||||
async with db_session() as db:
|
||||
h = pick(host, await _all_hosts(db), kind="host", list_tool="list_hosts")
|
||||
if not metric:
|
||||
latest = await metric_series.latest_values(db, AGENT_SOURCE, h.name)
|
||||
if not latest:
|
||||
raise ValueError(f"no agent metrics recorded for host {h.name!r}; "
|
||||
"is a host agent installed and reporting? list_hosts shows `agent`.")
|
||||
return {
|
||||
"host_id": h.id, "host_name": h.name,
|
||||
"metrics": [
|
||||
{"metric": s.metric_name, "last": rnd(s.value),
|
||||
"last_at": iso(s.recorded_at), "age_s": age_s(s.recorded_at)}
|
||||
for s in latest
|
||||
],
|
||||
"next": "Pass one of these as `metric` for its history.",
|
||||
}
|
||||
pts = await metric_series.points(db, AGENT_SOURCE, h.name, metric, since)
|
||||
if not pts:
|
||||
names = [s.metric_name for s in await metric_series.latest_values(db, AGENT_SOURCE, h.name)]
|
||||
if metric not in names:
|
||||
hint = (f" It reports: {', '.join(names)}." if names
|
||||
else " No agent metrics are recorded for it at all.")
|
||||
raise ValueError(f"host {h.name!r} has no metric {metric!r}.{hint}")
|
||||
return {
|
||||
"host_id": h.id, "host_name": h.name, "metric": metric, "range": key,
|
||||
**summarize(pts),
|
||||
}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(list_hosts, role=UserRole.viewer, read_only=True)
|
||||
register_tool(get_host, role=UserRole.viewer, read_only=True)
|
||||
register_tool(get_host_metrics, role=UserRole.viewer, read_only=True)
|
||||
@@ -1,71 +0,0 @@
|
||||
"""Generic metric access: anything recorded in Steward's metric store."""
|
||||
from __future__ import annotations
|
||||
|
||||
from steward.core import metric_series
|
||||
from steward.models.users import UserRole
|
||||
|
||||
from ._shape import age_s, bounded, clamp_limit, db_session, iso, lookback, rnd, summarize
|
||||
|
||||
|
||||
async def query_metrics(source_module: str = "", resource: str = "", metric: str = "",
|
||||
range: str = "24h", limit: int = 200) -> dict:
|
||||
"""Read Steward's metric store, narrowing one argument at a time.
|
||||
|
||||
Every plugin and monitor type records into one store keyed by
|
||||
(source_module, resource, metric): host_agent, docker, traefik, unifi,
|
||||
snmp, the monitor types (icmp/tcp/dns/http) and Steward's own self-monitor.
|
||||
|
||||
- No arguments: the source modules that have data.
|
||||
- source_module only: its resources (hosts, routers, containers, devices).
|
||||
A host agent's sub-resources are named "<host>:<part>", e.g. "nas:/".
|
||||
- source_module + resource: the latest value of each metric it reports.
|
||||
- all three: that metric over `range` ("1h", "6h", "24h" default, "7d",
|
||||
"30d") as min/avg/max/p95/last plus a ≤60-point [time, value] series.
|
||||
|
||||
limit bounds the resource list (default 200, max 1000).
|
||||
"""
|
||||
limit = clamp_limit(limit, 200, 1000)
|
||||
async with db_session() as db:
|
||||
if not source_module:
|
||||
return {"source_modules": await metric_series.sources(db),
|
||||
"next": "Pass one as `source_module` to list its resources."}
|
||||
|
||||
if not resource:
|
||||
names = await metric_series.resources(db, source_module)
|
||||
if not names:
|
||||
known = await metric_series.sources(db)
|
||||
raise ValueError(f"no metrics recorded for source_module {source_module!r}. "
|
||||
f"Sources with data: {', '.join(known) or 'none'}.")
|
||||
return {"source_module": source_module,
|
||||
**bounded(names, limit, key="resources",
|
||||
more="Pass a larger `limit` (max 1000)."),
|
||||
"next": "Pass one as `resource` to see its metrics."}
|
||||
|
||||
if not metric:
|
||||
latest = await metric_series.latest_values(db, source_module, resource)
|
||||
if not latest:
|
||||
raise ValueError(f"no metrics for resource {resource!r} in {source_module!r}; "
|
||||
"call query_metrics with only source_module to list its resources.")
|
||||
return {
|
||||
"source_module": source_module, "resource": resource,
|
||||
"metrics": [
|
||||
{"metric": s.metric_name, "last": rnd(s.value),
|
||||
"last_at": iso(s.recorded_at), "age_s": age_s(s.recorded_at)}
|
||||
for s in latest
|
||||
],
|
||||
"next": "Pass one as `metric` for its history.",
|
||||
}
|
||||
|
||||
since, key = lookback(range)
|
||||
pts = await metric_series.points(db, source_module, resource, metric, since)
|
||||
if not pts:
|
||||
names = [s.metric_name for s in await metric_series.latest_values(db, source_module, resource)]
|
||||
if metric not in names:
|
||||
raise ValueError(f"{resource!r} in {source_module!r} has no metric {metric!r}. "
|
||||
f"It reports: {', '.join(names) or 'nothing'}.")
|
||||
return {"source_module": source_module, "resource": resource, "metric": metric,
|
||||
"range": key, **summarize(pts)}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(query_metrics, role=UserRole.viewer, read_only=True)
|
||||
@@ -1,276 +0,0 @@
|
||||
"""Monitor tools: every uptime check, and one check's failures in depth."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.models.alerts import AlertRule
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.monitors import MONITOR_TYPE_VALUES, Monitor
|
||||
from steward.models.users import UserRole
|
||||
from steward.monitors import service as monitor_svc
|
||||
|
||||
from ._audit import audit_mcp
|
||||
from ._shape import (
|
||||
age_s, bounded, clamp_limit, db_session, iso, now_utc, pick, redact_url, rnd,
|
||||
)
|
||||
|
||||
_STATES = ("up", "down", "pending", "disabled")
|
||||
|
||||
|
||||
def _state(m: Monitor, latest) -> str:
|
||||
if not m.enabled:
|
||||
return "disabled"
|
||||
if latest is None:
|
||||
return "pending"
|
||||
return "up" if latest.is_up else "down"
|
||||
|
||||
|
||||
def _tls_days(latest) -> float | None:
|
||||
if latest is None or latest.tls_expires_at is None:
|
||||
return None
|
||||
return rnd((latest.tls_expires_at - now_utc()).total_seconds() / 86400.0, 1)
|
||||
|
||||
|
||||
def _row(m: Monitor, st: dict, host_names: dict[str, str]) -> dict:
|
||||
latest = st["latest"]
|
||||
return {
|
||||
"monitor_id": m.id,
|
||||
"monitor_name": m.name,
|
||||
"type": m.type,
|
||||
"target": redact_url(m.target),
|
||||
"host_id": m.host_id,
|
||||
"host_name": host_names.get(m.host_id) if m.host_id else None,
|
||||
"state": _state(m, latest),
|
||||
"down_since": iso(st["down_since"]),
|
||||
"down_for_s": age_s(st["down_since"]),
|
||||
"last_checked_at": iso(latest.checked_at) if latest else None,
|
||||
"response_ms": rnd(latest.response_ms) if latest else None,
|
||||
"error": latest.error_msg if latest is not None and not latest.is_up else None,
|
||||
"uptime_pct": st["uptime"] or {"24h": None, "7d": None, "30d": None},
|
||||
"tls_expires_in_days": _tls_days(latest),
|
||||
}
|
||||
|
||||
|
||||
async def _load(db):
|
||||
monitors = list((await db.execute(select(Monitor).order_by(Monitor.name))).scalars())
|
||||
host_names = dict((await db.execute(select(Host.id, Host.name))).all())
|
||||
return monitors, host_names
|
||||
|
||||
|
||||
async def list_monitors(state: str = "", type: str = "", limit: int = 100) -> dict:
|
||||
"""Every uptime monitor (ping, TCP, DNS, HTTP) with its current state.
|
||||
|
||||
state: "down", "up", "pending" (never checked yet) or "disabled"; empty for all.
|
||||
type: "icmp", "tcp", "dns" or "http"; empty for all.
|
||||
limit: default 100, max 500.
|
||||
|
||||
Down monitors come first, each with `down_since` (when its current run of
|
||||
failures began) and the latest `error`. `uptime_pct` covers 24h/7d/30d;
|
||||
`tls_expires_in_days` is set for HTTPS checks. get_monitor shows one
|
||||
monitor's configuration and recent failures.
|
||||
"""
|
||||
if state and state not in _STATES:
|
||||
raise ValueError(f"state {state!r} is not one of {', '.join(_STATES)}.")
|
||||
if type and type not in MONITOR_TYPE_VALUES:
|
||||
raise ValueError(f"type {type!r} is not one of {', '.join(MONITOR_TYPE_VALUES)}.")
|
||||
limit = clamp_limit(limit, 100, 500)
|
||||
async with db_session() as db:
|
||||
monitors, host_names = await _load(db)
|
||||
if type:
|
||||
monitors = [m for m in monitors if m.type == type]
|
||||
states = await monitor_svc.monitor_states(db, monitors)
|
||||
rows = [_row(m, states[m.id], host_names) for m in monitors]
|
||||
if state:
|
||||
rows = [r for r in rows if r["state"] == state]
|
||||
order = {"down": 0, "pending": 1, "up": 2, "disabled": 3}
|
||||
rows.sort(key=lambda r: (order[r["state"]], r["monitor_name"].lower()))
|
||||
counts = {s: 0 for s in _STATES}
|
||||
for r in rows:
|
||||
counts[r["state"]] += 1
|
||||
return {"counts": counts, **bounded(rows, limit, key="monitors",
|
||||
more="Pass a larger `limit`, or filter by `state` or `type`.")}
|
||||
|
||||
|
||||
def _config(m: Monitor) -> dict:
|
||||
try:
|
||||
cfg = json.loads(m.config_json or "{}")
|
||||
except ValueError:
|
||||
return {}
|
||||
if isinstance(cfg.get("headers"), dict):
|
||||
# Header values are often credentials (Authorization, API keys).
|
||||
cfg["headers"] = {k: "<redacted>" for k in cfg["headers"]}
|
||||
return cfg
|
||||
|
||||
|
||||
async def get_monitor(monitor: str, failures: int = 10) -> dict:
|
||||
"""One monitor in depth: configuration, current state, uptime and recent failures.
|
||||
|
||||
monitor: the monitor's name (case-insensitive) or id, as list_monitors shows.
|
||||
failures: how many of its latest failed checks to return (default 10, max 50),
|
||||
newest first, each with its error message.
|
||||
|
||||
`heartbeat` is its last 30 checks oldest-first as a string, "+" up and "-"
|
||||
down, to show a flapping check at a glance. Header values in an HTTP
|
||||
monitor's configuration are redacted.
|
||||
"""
|
||||
failures = clamp_limit(failures, 10, 50)
|
||||
async with db_session() as db:
|
||||
monitors, host_names = await _load(db)
|
||||
m = pick(monitor, monitors, kind="monitor", list_tool="list_monitors")
|
||||
st = (await monitor_svc.monitor_states(db, [m]))[m.id]
|
||||
failed = await monitor_svc.recent_failures(db, m.id, failures)
|
||||
return {
|
||||
**_row(m, st, host_names),
|
||||
"check_interval_s": m.check_interval_seconds or None,
|
||||
"config": _config(m),
|
||||
"heartbeat": "".join("+" if r.is_up else "-" for r in st["recent"]),
|
||||
"recent_failures": [
|
||||
{
|
||||
"checked_at": iso(r.checked_at),
|
||||
"age_s": age_s(r.checked_at),
|
||||
"error": r.error_msg,
|
||||
"status_code": r.status_code,
|
||||
"response_ms": rnd(r.response_ms),
|
||||
}
|
||||
for r in failed
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# ── changes ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _host_id(db, host: str) -> str | None:
|
||||
if not (host or "").strip():
|
||||
return None
|
||||
hosts = (await db.execute(select(Host))).scalars()
|
||||
return pick(host, hosts, kind="host", list_tool="list_hosts").id
|
||||
|
||||
|
||||
def _saved(m: Monitor, host_names: dict[str, str]) -> dict:
|
||||
return {
|
||||
"monitor_id": m.id,
|
||||
"monitor_name": m.name,
|
||||
"type": m.type,
|
||||
"target": redact_url(m.target),
|
||||
"host_name": host_names.get(m.host_id) if m.host_id else None,
|
||||
"enabled": m.enabled,
|
||||
"check_interval_s": m.check_interval_seconds or None,
|
||||
"config": _config(m),
|
||||
"next": f'get_monitor(monitor="{m.id}")',
|
||||
}
|
||||
|
||||
|
||||
async def create_monitor(name: str, type: str, target: str, host: str = "",
|
||||
config: dict | None = None, check_interval_s: int = 0,
|
||||
enabled: bool = True) -> dict:
|
||||
"""Add an uptime check. Recorded in the audit log as "<you> via MCP".
|
||||
|
||||
type: "icmp" (ping), "tcp" (connect to a port), "dns" (resolve a name)
|
||||
or "http" (request a URL).
|
||||
target: an address or hostname; for http a URL (a bare host gets https://).
|
||||
host: a Steward host's name or id to file the check under; empty for a
|
||||
free-standing destination.
|
||||
config, by type (every key optional, defaults shown):
|
||||
tcp: {"port": 80}
|
||||
dns: {"expected_ip": null} (set to fail when the name resolves elsewhere)
|
||||
http: {"method": "GET" | "HEAD" | "POST", "expected_status": 200,
|
||||
"content_match": "", "headers": {}, "timeout_seconds": 10 (1-60),
|
||||
"follow_redirects": true, "verify_ssl": true}
|
||||
icmp takes none.
|
||||
check_interval_s: seconds between checks; 0 (default) uses the global
|
||||
interval from Settings.
|
||||
|
||||
The check's metrics are reported under source_module = its type and
|
||||
resource = its name, which is what an alert rule on it watches.
|
||||
"""
|
||||
fields = monitor_svc.monitor_fields(type, target=target, name=name,
|
||||
check_interval_seconds=check_interval_s, opts=config or {})
|
||||
async with db_session() as db, db.begin():
|
||||
m = Monitor(host_id=await _host_id(db, host), enabled=enabled, **fields)
|
||||
db.add(m)
|
||||
_monitors, host_names = await _load(db)
|
||||
await audit_mcp("monitor.created", "monitor", m.name,
|
||||
{"monitor_id": m.id, "type": m.type, "target": redact_url(m.target)})
|
||||
return _saved(m, host_names)
|
||||
|
||||
|
||||
async def update_monitor(monitor: str, name: str | None = None, target: str | None = None,
|
||||
host: str | None = None, config: dict | None = None,
|
||||
check_interval_s: int | None = None) -> dict:
|
||||
"""Change a monitor. Only the arguments you pass change; the rest stay.
|
||||
|
||||
monitor: its name or id (list_monitors).
|
||||
host: a host's name or id to file it under, or "" to unlink it.
|
||||
config: the keys to change, merged over its current config (see
|
||||
create_monitor for the keys of each type). A monitor's type cannot
|
||||
change; delete it and create another.
|
||||
|
||||
Alert rules watch a monitor by its name, so a rename returns `warning`
|
||||
naming the rules still on the old name.
|
||||
|
||||
Recorded in the audit log.
|
||||
"""
|
||||
async with db_session() as db, db.begin():
|
||||
monitors, host_names = await _load(db)
|
||||
m = pick(monitor, monitors, kind="monitor", list_tool="list_monitors")
|
||||
current = {k: v for k, v in m.config.items() if k in monitor_svc.CONFIG_KEYS[m.type]}
|
||||
fields = monitor_svc.monitor_fields(
|
||||
m.type, target=m.target if target is None else target,
|
||||
name=m.name if name is None else name,
|
||||
check_interval_seconds=(m.check_interval_seconds if check_interval_s is None
|
||||
else check_interval_s),
|
||||
opts={**current, **(config or {})})
|
||||
old_name = m.name
|
||||
for k, v in fields.items():
|
||||
setattr(m, k, v)
|
||||
if host is not None:
|
||||
m.host_id = await _host_id(db, host)
|
||||
stranded = []
|
||||
if m.name != old_name:
|
||||
stranded = list((await db.execute(
|
||||
select(AlertRule.name).where(AlertRule.source_module == m.type,
|
||||
AlertRule.resource_name == old_name))).scalars())
|
||||
changed = sorted(k for k, v in {"name": name, "target": target, "host": host, "config": config,
|
||||
"check_interval_s": check_interval_s}.items() if v is not None)
|
||||
await audit_mcp("monitor.updated", "monitor", m.name, {"monitor_id": m.id, "changed": changed})
|
||||
out = _saved(m, host_names)
|
||||
if stranded:
|
||||
out["warning"] = (f"Alert rules watch this monitor by name and still watch {old_name!r}: "
|
||||
f"{', '.join(stranded)}. update_alert_rule(resource=\"{m.name}\") "
|
||||
"points each at the new name.")
|
||||
return out
|
||||
|
||||
|
||||
async def set_monitor_enabled(monitor: str, enabled: bool) -> dict:
|
||||
"""Pause (enabled=false) or resume a monitor. A paused monitor runs no checks
|
||||
and raises no alerts. Recorded in the audit log."""
|
||||
async with db_session() as db, db.begin():
|
||||
monitors, host_names = await _load(db)
|
||||
m = pick(monitor, monitors, kind="monitor", list_tool="list_monitors")
|
||||
m.enabled = bool(enabled)
|
||||
await audit_mcp("monitor.enabled" if m.enabled else "monitor.disabled", "monitor", m.name,
|
||||
{"monitor_id": m.id})
|
||||
return _saved(m, host_names)
|
||||
|
||||
|
||||
async def delete_monitor(monitor: str) -> dict:
|
||||
"""Delete a monitor and its check history. To stop it but keep its history,
|
||||
set_monitor_enabled(enabled=false) instead. Admin only; recorded in the
|
||||
audit log."""
|
||||
async with db_session() as db, db.begin():
|
||||
monitors, _host_names = await _load(db)
|
||||
m = pick(monitor, monitors, kind="monitor", list_tool="list_monitors")
|
||||
await db.delete(m)
|
||||
await audit_mcp("monitor.deleted", "monitor", m.name,
|
||||
{"monitor_id": m.id, "type": m.type, "target": redact_url(m.target)})
|
||||
return {"deleted": m.name, "monitor_id": m.id}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(list_monitors, role=UserRole.viewer, read_only=True)
|
||||
register_tool(get_monitor, role=UserRole.viewer, read_only=True)
|
||||
for fn in (create_monitor, update_monitor, set_monitor_enabled):
|
||||
register_tool(fn, role=UserRole.operator, read_only=False)
|
||||
register_tool(delete_monitor, role=UserRole.admin, read_only=False)
|
||||
@@ -1,225 +0,0 @@
|
||||
"""get_overview: what is wrong right now, ranked, across everything Steward watches."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.hosts import service as host_svc
|
||||
from steward.mcp._context import mcp_app
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.monitors import Monitor
|
||||
from steward.models.users import UserRole
|
||||
from steward.monitors import service as monitor_svc
|
||||
|
||||
from ._shape import age_s, db_session, iso, now_utc, rnd
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FINDINGS_CAP = 50
|
||||
_SEVERITY = {"critical": 0, "warning": 1}
|
||||
|
||||
|
||||
def finding(severity: str, kind: str, subject: str, summary: str, *, since=None,
|
||||
ids: dict | None = None, next: str | None = None) -> dict:
|
||||
"""One overview finding. Plugins' `mcp_overview(db)` return a list of these."""
|
||||
return {
|
||||
"severity": severity, "kind": kind, "subject": subject, "summary": summary,
|
||||
"since": iso(since), "age_s": age_s(since), "ids": ids or {}, "next": next,
|
||||
}
|
||||
|
||||
|
||||
def rank(findings: list[dict]) -> list[dict]:
|
||||
"""Critical first; within a severity, the longest-standing first."""
|
||||
return sorted(findings, key=lambda f: (_SEVERITY.get(f.get("severity"), 2),
|
||||
-(f.get("age_s") or 0), f.get("subject", "")))
|
||||
|
||||
|
||||
def monitor_findings(monitors, states: dict, host_names: dict, now,
|
||||
tls: dict | None = None) -> list[dict]:
|
||||
"""Failing monitors, and HTTPS certificates inside the TLS window
|
||||
(`tls` is THRESHOLDS["tls_days"]: warn/crit days left)."""
|
||||
tls = tls or {"warn": 14, "crit": 3}
|
||||
out = []
|
||||
for m in monitors:
|
||||
if not m.enabled:
|
||||
continue
|
||||
st = states.get(m.id) or {}
|
||||
latest = st.get("latest")
|
||||
if latest is None:
|
||||
continue
|
||||
where = f" on {host_names[m.host_id]}" if m.host_id in host_names else ""
|
||||
ids = {"monitor_id": m.id, "host_id": m.host_id}
|
||||
if not latest.is_up:
|
||||
out.append(finding(
|
||||
"critical", "monitor_down", m.name,
|
||||
f"{m.type} check of {m.target}{where} is failing: {latest.error_msg or 'down'}",
|
||||
since=st.get("down_since") or latest.checked_at, ids=ids,
|
||||
next=f'get_monitor(monitor="{m.id}")'))
|
||||
elif latest.tls_expires_at is not None:
|
||||
days = (latest.tls_expires_at - now).total_seconds() / 86400.0
|
||||
if days <= tls["warn"]:
|
||||
out.append(finding(
|
||||
"critical" if days <= tls["crit"] else "warning", "tls_expiring", m.name,
|
||||
f"TLS certificate for {m.target} expires in {days:.1f} days",
|
||||
ids=ids, next=f'get_monitor(monitor="{m.id}")'))
|
||||
return out
|
||||
|
||||
|
||||
def alert_findings(active) -> list[dict]:
|
||||
out = []
|
||||
for state, rule in active:
|
||||
s = state.state.value
|
||||
if s == "firing":
|
||||
sev, what = "critical", "is firing"
|
||||
elif s == "pending":
|
||||
sev, what = "warning", "is pending (breaching, not yet for long enough to fire)"
|
||||
else:
|
||||
sev, what = "warning", "is acknowledged but still breaching"
|
||||
out.append(finding(
|
||||
sev, f"alert_{s}", rule.name,
|
||||
f"{rule.metric_name} {rule.operator.value} {rule.threshold:g} on "
|
||||
f"{rule.source_module}/{rule.resource_name} {what}",
|
||||
since=state.last_transition_at, ids={"rule_id": rule.id},
|
||||
next=f'get_alert_history(rule="{rule.id}")'))
|
||||
return out
|
||||
|
||||
|
||||
_AGENT_CHECKS = (("cpu_pct", "cpu", "CPU"), ("mem_used_pct", "mem", "memory"),
|
||||
("disk_used_pct_worst", "disk", "fullest disk"))
|
||||
|
||||
|
||||
def agent_findings(hosts, agent: dict, thresholds: dict) -> list[dict]:
|
||||
out = []
|
||||
for h in hosts:
|
||||
a = agent.get(h.name)
|
||||
if not a:
|
||||
continue
|
||||
ids = {"host_id": h.id}
|
||||
if not a.get("fresh"):
|
||||
out.append(finding(
|
||||
"warning", "agent_stale", h.name,
|
||||
"host agent has stopped reporting; its metrics are out of date",
|
||||
since=a.get("last_at"), ids=ids, next=f'get_host(host="{h.id}")'))
|
||||
continue # stale values would only add noise
|
||||
for metric, kind, label in _AGENT_CHECKS:
|
||||
value = a.get(metric)
|
||||
th = thresholds.get(kind) or {}
|
||||
if value is None or th.get("warn") is None:
|
||||
continue
|
||||
if th.get("crit") is not None and value >= th["crit"]:
|
||||
sev = "critical"
|
||||
elif value >= th["warn"]:
|
||||
sev = "warning"
|
||||
else:
|
||||
continue
|
||||
out.append(finding(
|
||||
sev, f"host_{kind}_high", h.name,
|
||||
f"{label} at {rnd(value, 1)}% (warn {th['warn']}%, crit {th.get('crit')}%)",
|
||||
ids=ids, next=f'get_host_metrics(host="{h.id}", metric="{metric}")'))
|
||||
return out
|
||||
|
||||
|
||||
def run_findings(runs) -> list[dict]:
|
||||
out = []
|
||||
for r in runs:
|
||||
failures = ((r.results or {}).get("failures") or [])[:3]
|
||||
detail = f": {'; '.join(str(f)[:200] for f in failures)}" if failures else ""
|
||||
out.append(finding(
|
||||
"warning", f"ansible_run_{r.status.value}", r.playbook_path,
|
||||
f"Ansible run of {r.playbook_path} ({r.source_name}) {r.status.value}{detail}",
|
||||
since=r.finished_at or r.started_at, ids={"run_id": r.id}))
|
||||
return out
|
||||
|
||||
|
||||
async def _plugin_findings(server) -> tuple[list[dict], list[str]]:
|
||||
found, unavailable = [], []
|
||||
for plugin, source in sorted((getattr(server, "overview_sources", None) or {}).items()):
|
||||
if not server.plugin_enabled(plugin):
|
||||
continue
|
||||
try:
|
||||
async with db_session() as db:
|
||||
found.extend(await source(db))
|
||||
except Exception:
|
||||
logger.exception("MCP overview source of plugin %r failed", plugin)
|
||||
unavailable.append(plugin)
|
||||
return found, unavailable
|
||||
|
||||
|
||||
async def get_overview() -> dict:
|
||||
"""What is wrong right now, across everything Steward watches. Start here.
|
||||
|
||||
Returns `findings` ranked critical first, then longest-standing first:
|
||||
failing monitors (with the error and since when), firing, pending and
|
||||
still-breaching acknowledged alerts, TLS certificates near expiry, host
|
||||
agents that stopped reporting, hosts over their CPU, memory or disk
|
||||
limits (all cutoffs are set under Settings → Thresholds), Ansible runs that failed
|
||||
in the last 24h, and anything enabled plugins add (unhealthy containers,
|
||||
for one). Each finding carries the ids to follow and `next`, the call
|
||||
that shows its evidence.
|
||||
|
||||
`counts` sizes the fleet, and `active_maintenance` names windows
|
||||
currently silencing alerts. `all_clear` is true when nothing is wrong.
|
||||
"""
|
||||
app = mcp_app()
|
||||
now = now_utc()
|
||||
async with db_session() as db:
|
||||
hosts = list((await db.execute(select(Host).order_by(Host.name))).scalars())
|
||||
host_names = {h.id: h.name for h in hosts}
|
||||
monitors = list((await db.execute(select(Monitor).order_by(Monitor.name))).scalars())
|
||||
states = await monitor_svc.monitor_states(db, [m for m in monitors if m.enabled])
|
||||
active = await alert_svc.active_alerts(db)
|
||||
agent = await host_svc.agent_overview_by_host(
|
||||
db, [h.name for h in hosts], host_svc.agent_stale_after_seconds(app.config))
|
||||
runs = list((await db.execute(
|
||||
select(AnsibleRun)
|
||||
.where(AnsibleRun.status.in_((AnsibleRunStatus.failed, AnsibleRunStatus.interrupted)),
|
||||
AnsibleRun.started_at >= now - timedelta(hours=24))
|
||||
.order_by(AnsibleRun.started_at.desc())
|
||||
.limit(20)
|
||||
)).scalars())
|
||||
windows = await alert_svc.list_maintenance_windows(db)
|
||||
|
||||
thresholds = app.config.get("THRESHOLDS") or {}
|
||||
findings = (monitor_findings(monitors, states, host_names, now, thresholds.get("tls_days"))
|
||||
+ alert_findings(active)
|
||||
+ agent_findings(hosts, agent, thresholds)
|
||||
+ run_findings(runs))
|
||||
plugin_found, unavailable = await _plugin_findings(getattr(app, "mcp_server", None))
|
||||
findings = rank(findings + plugin_found)
|
||||
|
||||
down = sum(1 for m in monitors if m.enabled and (st := states.get(m.id))
|
||||
and st["latest"] is not None and not st["latest"].is_up)
|
||||
out = {
|
||||
"generated_at": iso(now),
|
||||
"all_clear": not findings,
|
||||
"counts": {
|
||||
"hosts": len(hosts),
|
||||
"monitors": len(monitors),
|
||||
"monitors_down": down,
|
||||
"alerts_active": len(active),
|
||||
"critical": sum(1 for f in findings if f["severity"] == "critical"),
|
||||
"warning": sum(1 for f in findings if f["severity"] == "warning"),
|
||||
},
|
||||
"active_maintenance": [
|
||||
{"window_id": w.id, "name": w.name, "end_at": iso(w.end_at),
|
||||
"scope": {"source_module": w.scope_source, "resource": w.scope_resource}}
|
||||
for w in windows if w.start_at <= now <= w.end_at
|
||||
],
|
||||
"findings": findings[:FINDINGS_CAP],
|
||||
"truncated": len(findings) > FINDINGS_CAP,
|
||||
}
|
||||
if out["truncated"]:
|
||||
out["more"] = (f"{len(findings) - FINDINGS_CAP} lower-ranked findings left out; "
|
||||
"list_monitors(state=\"down\"), list_alerts and list_hosts show everything.")
|
||||
if unavailable:
|
||||
out["unavailable"] = [f"{p} plugin's findings could not be read (see Steward's log)"
|
||||
for p in unavailable]
|
||||
return out
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(get_overview, role=UserRole.viewer, read_only=True)
|
||||
@@ -1,35 +0,0 @@
|
||||
"""Session tools: who the connected client is acting as."""
|
||||
from __future__ import annotations
|
||||
|
||||
from steward.mcp._context import current_caller
|
||||
from steward.models.users import UserRole
|
||||
|
||||
_ROLE_MEANS = {
|
||||
UserRole.viewer: "Read-only: every list_*/get_* tool. Nothing that changes Steward or your hosts.",
|
||||
UserRole.operator: "Viewer tools, plus running playbooks (check mode unless told otherwise) "
|
||||
"and changing monitors, alerts and schedules.",
|
||||
UserRole.admin: "Everything, including the playbook an alert rule runs, pruning Docker disk, "
|
||||
"and deleting monitors and alert rules.",
|
||||
}
|
||||
|
||||
|
||||
async def whoami() -> dict:
|
||||
"""Who this connection acts as in Steward, and what that role may do.
|
||||
|
||||
Call this first if a tool refuses you for lack of role: `role` is the
|
||||
LOWER of the token's ceiling and its user's current role, so it can be
|
||||
less than the token was created with. A higher role needs a new token
|
||||
from Steward's AI access page, and only up to the user's own role.
|
||||
"""
|
||||
caller = current_caller()
|
||||
return {
|
||||
"user": caller.username,
|
||||
"user_id": caller.user_id,
|
||||
"role": caller.role.value,
|
||||
"role_allows": _ROLE_MEANS[caller.role],
|
||||
"token_id": caller.token_id,
|
||||
}
|
||||
|
||||
|
||||
def register(register_tool) -> None:
|
||||
register_tool(whoami, role=UserRole.viewer, read_only=True)
|
||||
@@ -1,40 +0,0 @@
|
||||
"""Drop the stored setting for the removed http plugin
|
||||
|
||||
The standalone `http` plugin was dissolved into the unified Monitor entity
|
||||
(0022_unify_monitors), but its `plugin.http` settings key outlived it: the key
|
||||
stayed in core DEFAULTS, so the operator saw a plugin that no longer exists
|
||||
reported as "enabled" with no way to clear it — deleting the row alone did
|
||||
nothing, because the default reasserted it on the next settings load.
|
||||
|
||||
The DEFAULTS entry is removed in the same change; this migration clears any row
|
||||
an operator's database still carries so the two agree. Per family rule 22, the
|
||||
removed subsystem takes its setting row with it.
|
||||
|
||||
Note: `http` remains a valid MONITOR TYPE (icmp/tcp/dns/http). This touches only
|
||||
the plugin-enablement key, never monitor data.
|
||||
|
||||
Revision ID: 0025_drop_http_plugin_setting
|
||||
Revises: 0024_plugin_metrics_hourly
|
||||
Create Date: 2026-08-13
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "0025_drop_http_plugin_setting"
|
||||
down_revision: Union[str, None] = "0024_plugin_metrics_hourly"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
sa.text("DELETE FROM app_settings WHERE key = 'plugin.http'")
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Deliberately empty. Re-inserting `plugin.http` would recreate the exact
|
||||
# phantom this migration exists to remove, and the plugin it configured no
|
||||
# longer exists to read it.
|
||||
pass
|
||||
@@ -1,41 +0,0 @@
|
||||
"""Personal API tokens for MCP clients
|
||||
|
||||
Revision ID: 0026_api_tokens
|
||||
Revises: 0025_drop_http_plugin_setting
|
||||
Create Date: 2026-10-06
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision: str = "0026_api_tokens"
|
||||
down_revision: Union[str, None] = "0025_drop_http_plugin_setting"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"api_tokens",
|
||||
sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
|
||||
sa.Column("user_id", sa.String(36),
|
||||
sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False),
|
||||
sa.Column("name", sa.String(128), nullable=False),
|
||||
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
|
||||
sa.Column("prefix", sa.String(16), nullable=False),
|
||||
# Reuses the users.role enum type created in 0001.
|
||||
sa.Column("role", postgresql.ENUM("admin", "operator", "viewer",
|
||||
name="userrole", create_type=False),
|
||||
nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("last_used_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
op.create_index("ix_api_tokens_user_id", "api_tokens", ["user_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_api_tokens_user_id", "api_tokens")
|
||||
op.drop_table("api_tokens")
|
||||
@@ -1,6 +1,5 @@
|
||||
from .base import Base
|
||||
from .users import User
|
||||
from .api_tokens import ApiToken
|
||||
from .hosts import Host
|
||||
from .monitors import Monitor, MonitorResult, MonitorType
|
||||
from .metrics import PluginMetric
|
||||
@@ -12,7 +11,7 @@ from .settings import AppSetting
|
||||
from .dashboard import Dashboard, DashboardWidget, DashboardShareToken
|
||||
|
||||
__all__ = [
|
||||
"Base", "User", "ApiToken",
|
||||
"Base", "User",
|
||||
"Host",
|
||||
"Monitor", "MonitorResult", "MonitorType",
|
||||
"PluginMetric",
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
from __future__ import annotations
|
||||
from datetime import datetime, timezone
|
||||
from sqlalchemy import DateTime, Enum, ForeignKey, Integer, String
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from .base import Base
|
||||
from .users import UserRole
|
||||
|
||||
|
||||
class ApiToken(Base):
|
||||
"""A personal bearer token an AI client (MCP) uses to act as its user.
|
||||
|
||||
Only the sha256 of the token is stored; the plaintext is shown once at
|
||||
creation. `role` is a ceiling: the token acts with the lower of this and
|
||||
its user's current role, so demoting a user demotes their tokens too.
|
||||
"""
|
||||
__tablename__ = "api_tokens"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
user_id: Mapped[str] = mapped_column(
|
||||
String(36), ForeignKey("users.id", ondelete="CASCADE"), nullable=False,
|
||||
)
|
||||
name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
token_hash: Mapped[str] = mapped_column(String(64), nullable=False, unique=True)
|
||||
# Leading characters of the plaintext, for telling tokens apart in the UI.
|
||||
prefix: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
role: Mapped[UserRole] = mapped_column(
|
||||
Enum(UserRole, name="userrole", create_type=False), nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False,
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
)
|
||||
last_used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
+83
-36
@@ -5,38 +5,86 @@ type-aware add/edit form, one dashboard widget. A monitor may be linked to a
|
||||
Host (host_id) or watch a free-standing custom destination (host_id = NULL).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from quart import Blueprint, current_app, render_template, request, redirect, url_for
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.auth.middleware import require_role
|
||||
from steward.models.users import UserRole
|
||||
from steward.models.monitors import Monitor, MonitorType, MONITOR_TYPE_VALUES
|
||||
from steward.core.time_range import DEFAULT_RANGE
|
||||
from steward.monitors import service as monitor_svc
|
||||
from steward.core.status import _last_n_results, _uptime_by_monitor, _heartbeat_title
|
||||
from steward.core.time_range import parse_range, DEFAULT_RANGE
|
||||
|
||||
monitors_bp = Blueprint("monitors", __name__, url_prefix="/monitors")
|
||||
|
||||
|
||||
def _form_opts(form, mtype: str) -> dict:
|
||||
"""The submitted fields of `mtype`'s config, for monitor_svc.build_config.
|
||||
An unchecked checkbox is absent from the form, so it reads as False."""
|
||||
def _build_config(form, mtype: str) -> dict:
|
||||
"""Assemble the type-specific config dict from submitted form fields."""
|
||||
if mtype == MonitorType.tcp.value:
|
||||
return {"port": form.get("port")}
|
||||
return {"port": int(form.get("port") or 80)}
|
||||
if mtype == MonitorType.dns.value:
|
||||
return {"expected_ip": form.get("expected_ip")}
|
||||
return {"expected_ip": (form.get("expected_ip", "").strip() or None)}
|
||||
if mtype == MonitorType.http.value:
|
||||
try:
|
||||
headers = json.loads(form.get("headers_json") or "{}")
|
||||
except (ValueError, TypeError):
|
||||
headers = {}
|
||||
return {
|
||||
"method": form.get("method"),
|
||||
"expected_status": form.get("expected_status"),
|
||||
"content_match": form.get("content_match"),
|
||||
"headers": form.get("headers_json"),
|
||||
"timeout_seconds": form.get("timeout_seconds"),
|
||||
"method": form.get("method", "GET").upper(),
|
||||
"expected_status": int(form.get("expected_status", 200) or 200),
|
||||
"content_match": form.get("content_match", "").strip(),
|
||||
"headers": headers,
|
||||
"timeout_seconds": int(form.get("timeout_seconds", 10) or 10),
|
||||
"follow_redirects": "follow_redirects" in form,
|
||||
"verify_ssl": "verify_ssl" in form,
|
||||
}
|
||||
return {}
|
||||
return {} # icmp
|
||||
|
||||
|
||||
def _normalise_target(mtype: str, target: str) -> str:
|
||||
"""For http, default a bare host to https://; others pass through."""
|
||||
target = target.strip()
|
||||
if mtype == MonitorType.http.value and target and not target.startswith(("http://", "https://")):
|
||||
return "https://" + target
|
||||
return target
|
||||
|
||||
|
||||
async def _rows_data(range_key: str):
|
||||
"""Per-monitor display data for the list / widget."""
|
||||
since, range_key = parse_range(range_key)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
monitors = list((await db.execute(
|
||||
select(Monitor).order_by(Monitor.name)
|
||||
)).scalars())
|
||||
ids = [m.id for m in monitors]
|
||||
recent = await _last_n_results(db, ids)
|
||||
uptime = await _uptime_by_monitor(db, ids)
|
||||
|
||||
# Map the selected range to the uptime bucket we surface in the list.
|
||||
bucket = {"24h": "24h", "7d": "7d", "30d": "30d"}.get(range_key, "24h")
|
||||
now = datetime.now(timezone.utc)
|
||||
data = []
|
||||
for m in monitors:
|
||||
rows = recent.get(m.id, [])
|
||||
latest = rows[-1] if rows else None
|
||||
tls_days = None
|
||||
if latest and latest.tls_expires_at:
|
||||
tls_days = (latest.tls_expires_at - now).total_seconds() / 86400.0
|
||||
data.append({
|
||||
"monitor": m,
|
||||
"latest": latest,
|
||||
"heartbeat": [{"state": "up" if r.is_up else "down",
|
||||
"title": _heartbeat_title(m.type, r)} for r in rows],
|
||||
"uptime_pct": uptime.get(m.id, {}).get(bucket),
|
||||
"tls_days": tls_days,
|
||||
})
|
||||
up = sum(1 for d in data if d["latest"] and d["latest"].is_up)
|
||||
down = sum(1 for d in data if d["latest"] and not d["latest"].is_up)
|
||||
pending = sum(1 for d in data if not d["latest"])
|
||||
return data, up, down, pending, range_key
|
||||
|
||||
|
||||
@monitors_bp.get("/")
|
||||
@@ -54,9 +102,9 @@ async def index():
|
||||
@monitors_bp.get("/rows")
|
||||
@require_role(UserRole.viewer)
|
||||
async def rows():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
data, up, down, pending, range_key = await monitor_svc.monitor_rows(
|
||||
db, request.args.get("range", DEFAULT_RANGE))
|
||||
data, up, down, pending, range_key = await _rows_data(
|
||||
request.args.get("range", DEFAULT_RANGE)
|
||||
)
|
||||
return await render_template(
|
||||
"monitors/rows.html",
|
||||
monitor_data=data, up=up, down=down, pending=pending, range_key=range_key,
|
||||
@@ -68,17 +116,22 @@ async def rows():
|
||||
async def add_monitor():
|
||||
form = await request.form
|
||||
mtype = form.get("type", "").strip()
|
||||
try:
|
||||
fields = monitor_svc.monitor_fields(
|
||||
mtype, target=form.get("target", ""), name=form.get("name", ""),
|
||||
check_interval_seconds=form.get("check_interval_seconds"),
|
||||
opts=_form_opts(form, mtype))
|
||||
except ValueError as exc:
|
||||
return str(exc), 400
|
||||
target = _normalise_target(mtype, form.get("target", ""))
|
||||
if mtype not in MONITOR_TYPE_VALUES or not target:
|
||||
return redirect(url_for("monitors.index"))
|
||||
|
||||
host_id = form.get("host_id", "").strip() or None
|
||||
monitor = Monitor(id=str(uuid.uuid4()), host_id=host_id, enabled=True,
|
||||
created_at=datetime.now(timezone.utc), **fields)
|
||||
monitor = Monitor(
|
||||
id=str(uuid.uuid4()),
|
||||
name=form.get("name", "").strip() or target,
|
||||
type=mtype,
|
||||
target=target,
|
||||
host_id=host_id,
|
||||
config_json=json.dumps(_build_config(form, mtype)),
|
||||
enabled=True,
|
||||
check_interval_seconds=int(form.get("check_interval_seconds", 0) or 0),
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
db.add(monitor)
|
||||
@@ -112,15 +165,10 @@ async def edit_monitor(monitor_id: str):
|
||||
return redirect(url_for("monitors.index"))
|
||||
# Type is immutable on edit (changes the meaning of config/target);
|
||||
# to switch type, delete and re-add.
|
||||
try:
|
||||
fields = monitor_svc.monitor_fields(
|
||||
m.type, target=form.get("target", m.target), name=form.get("name", ""),
|
||||
check_interval_seconds=form.get("check_interval_seconds"),
|
||||
opts=_form_opts(form, m.type))
|
||||
except ValueError as exc:
|
||||
return str(exc), 400
|
||||
for k, v in fields.items():
|
||||
setattr(m, k, v)
|
||||
m.name = form.get("name", "").strip() or m.target
|
||||
m.target = _normalise_target(m.type, form.get("target", m.target))
|
||||
m.config_json = json.dumps(_build_config(form, m.type))
|
||||
m.check_interval_seconds = int(form.get("check_interval_seconds", 0) or 0)
|
||||
return redirect(url_for("monitors.index"))
|
||||
|
||||
|
||||
@@ -154,8 +202,7 @@ async def widget():
|
||||
limit = max(1, min(50, int(request.args.get("limit", 10) or 10)))
|
||||
widget_id = request.args.get("wid", "0")
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
data, up, down, pending, _ = await monitor_svc.monitor_rows(db, DEFAULT_RANGE)
|
||||
data, up, down, pending, _ = await _rows_data(DEFAULT_RANGE)
|
||||
if type_filter in MONITOR_TYPE_VALUES:
|
||||
data = [d for d in data if d["monitor"].type == type_filter]
|
||||
if show_down_only:
|
||||
|
||||
@@ -1,202 +0,0 @@
|
||||
"""Monitor queries and validation shared by the monitors pages and the MCP tools.
|
||||
|
||||
Each function takes an open session (or plain values), with no Quart request
|
||||
context, so the routes and the /mcp tools (which run outside Quart's
|
||||
pipeline) call the same code and refuse the same input for the same reasons.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from steward.core.status import _heartbeat_title, _last_n_results, _uptime_by_monitor
|
||||
from steward.core.time_range import parse_range
|
||||
from steward.models.monitors import MONITOR_TYPE_VALUES, Monitor, MonitorResult, MonitorType
|
||||
|
||||
HTTP_METHODS = ("GET", "HEAD", "POST")
|
||||
# The config keys each type takes (models/monitors.py documents their meaning).
|
||||
CONFIG_KEYS: dict[str, tuple[str, ...]] = {
|
||||
MonitorType.icmp.value: (),
|
||||
MonitorType.tcp.value: ("port",),
|
||||
MonitorType.dns.value: ("expected_ip",),
|
||||
MonitorType.http.value: ("method", "expected_status", "content_match", "headers",
|
||||
"timeout_seconds", "follow_redirects", "verify_ssl"),
|
||||
}
|
||||
|
||||
|
||||
def _whole(name: str, value, default: int, lo: int, hi: int | None = None) -> int:
|
||||
if value is None or value == "":
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
raise ValueError(f"{name} must be a whole number, got {value!r}.")
|
||||
try:
|
||||
n = int(value)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError(f"{name} must be a whole number, got {value!r}.") from None
|
||||
if n < lo or (hi is not None and n > hi):
|
||||
bounds = f"between {lo} and {hi}" if hi is not None else f"at least {lo}"
|
||||
raise ValueError(f"{name} must be {bounds}, got {n}.")
|
||||
return n
|
||||
|
||||
|
||||
def _headers(value) -> dict:
|
||||
if value is None or value == "":
|
||||
return {}
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except ValueError:
|
||||
raise ValueError("headers must be a JSON object of name: value strings.") from None
|
||||
if not isinstance(value, dict) or not all(isinstance(k, str) and isinstance(v, str)
|
||||
for k, v in value.items()):
|
||||
raise ValueError("headers must be an object of name: value strings.")
|
||||
return value
|
||||
|
||||
|
||||
def normalise_target(mtype: str, target: str) -> str:
|
||||
"""For http, default a bare host to https://; others pass through."""
|
||||
target = (target or "").strip()
|
||||
if mtype == MonitorType.http.value and target and not target.startswith(("http://", "https://")):
|
||||
return "https://" + target
|
||||
return target
|
||||
|
||||
|
||||
def build_config(mtype: str, opts: Mapping) -> dict:
|
||||
"""The type-specific config from already-typed or form-string values.
|
||||
|
||||
Missing keys take their defaults; a key the type does not take, or a
|
||||
value out of range, raises ValueError saying which and why.
|
||||
"""
|
||||
unknown = sorted(set(opts) - set(CONFIG_KEYS[mtype]))
|
||||
if unknown:
|
||||
allowed = ", ".join(CONFIG_KEYS[mtype]) or "no config"
|
||||
raise ValueError(f"a {mtype} monitor does not take {', '.join(unknown)}; it takes {allowed}.")
|
||||
if mtype == MonitorType.tcp.value:
|
||||
return {"port": _whole("port", opts.get("port"), 80, 1, 65535)}
|
||||
if mtype == MonitorType.dns.value:
|
||||
return {"expected_ip": (str(opts.get("expected_ip") or "").strip() or None)}
|
||||
if mtype == MonitorType.http.value:
|
||||
method = str(opts.get("method") or "GET").strip().upper()
|
||||
if method not in HTTP_METHODS:
|
||||
raise ValueError(f"method {method!r} is not one of {', '.join(HTTP_METHODS)}.")
|
||||
return {
|
||||
"method": method,
|
||||
"expected_status": _whole("expected_status", opts.get("expected_status"), 200, 100, 599),
|
||||
"content_match": str(opts.get("content_match") or "").strip(),
|
||||
"headers": _headers(opts.get("headers")),
|
||||
"timeout_seconds": _whole("timeout_seconds", opts.get("timeout_seconds"), 10, 1, 60),
|
||||
"follow_redirects": bool(opts.get("follow_redirects", True)),
|
||||
"verify_ssl": bool(opts.get("verify_ssl", True)),
|
||||
}
|
||||
return {} # icmp
|
||||
|
||||
|
||||
def monitor_fields(mtype: str, *, target: str, name: str = "", check_interval_seconds=0,
|
||||
opts: Mapping | None = None) -> dict:
|
||||
"""Validated column values for a new or edited monitor; ValueError says why not."""
|
||||
mtype = (mtype or "").strip().lower()
|
||||
if mtype not in MONITOR_TYPE_VALUES:
|
||||
raise ValueError(f"type {mtype!r} is not one of {', '.join(MONITOR_TYPE_VALUES)}.")
|
||||
target = normalise_target(mtype, target)
|
||||
if not target:
|
||||
raise ValueError("a monitor needs a target: an address, or a URL for http.")
|
||||
if len(target) > 2048:
|
||||
raise ValueError("target is longer than 2048 characters.")
|
||||
return {
|
||||
"type": mtype,
|
||||
"target": target,
|
||||
"name": ((name or "").strip() or target)[:128],
|
||||
"config_json": json.dumps(build_config(mtype, opts or {})),
|
||||
"check_interval_seconds": _whole("check_interval_seconds", check_interval_seconds, 0, 0),
|
||||
}
|
||||
|
||||
|
||||
async def monitor_rows(db, range_key: str):
|
||||
"""Per-monitor state: latest result, heartbeat strip, uptime and TLS days.
|
||||
|
||||
Returns (rows, up, down, pending, range_key). `range_key` picks the uptime
|
||||
window surfaced (24h/7d/30d; anything else falls back to 24h) and comes
|
||||
back normalised.
|
||||
"""
|
||||
_since, range_key = parse_range(range_key)
|
||||
monitors = list((await db.execute(select(Monitor).order_by(Monitor.name))).scalars())
|
||||
ids = [m.id for m in monitors]
|
||||
recent = await _last_n_results(db, ids)
|
||||
uptime = await _uptime_by_monitor(db, ids)
|
||||
|
||||
# Map the selected range to the uptime bucket we surface in the list.
|
||||
bucket = {"24h": "24h", "7d": "7d", "30d": "30d"}.get(range_key, "24h")
|
||||
now = datetime.now(timezone.utc)
|
||||
data = []
|
||||
for m in monitors:
|
||||
rows = recent.get(m.id, [])
|
||||
latest = rows[-1] if rows else None
|
||||
tls_days = None
|
||||
if latest and latest.tls_expires_at:
|
||||
tls_days = (latest.tls_expires_at - now).total_seconds() / 86400.0
|
||||
data.append({
|
||||
"monitor": m,
|
||||
"latest": latest,
|
||||
"heartbeat": [{"state": "up" if r.is_up else "down",
|
||||
"title": _heartbeat_title(m.type, r)} for r in rows],
|
||||
"uptime_pct": uptime.get(m.id, {}).get(bucket),
|
||||
"tls_days": tls_days,
|
||||
})
|
||||
up = sum(1 for d in data if d["latest"] and d["latest"].is_up)
|
||||
down = sum(1 for d in data if d["latest"] and not d["latest"].is_up)
|
||||
pending = sum(1 for d in data if not d["latest"])
|
||||
return data, up, down, pending, range_key
|
||||
|
||||
|
||||
async def down_since(db, monitor_id: str):
|
||||
"""When the monitor's current run of failures began, or None if its latest check is up.
|
||||
|
||||
The first failed check after the last success; if it has never succeeded,
|
||||
its first check.
|
||||
"""
|
||||
last_up = (await db.execute(
|
||||
select(func.max(MonitorResult.checked_at))
|
||||
.where(MonitorResult.monitor_id == monitor_id, MonitorResult.is_up.is_(True))
|
||||
)).scalar()
|
||||
stmt = select(func.min(MonitorResult.checked_at)).where(MonitorResult.monitor_id == monitor_id)
|
||||
if last_up is not None:
|
||||
stmt = stmt.where(MonitorResult.checked_at > last_up)
|
||||
return (await db.execute(stmt)).scalar()
|
||||
|
||||
|
||||
async def monitor_states(db, monitors: list[Monitor]) -> dict[str, dict]:
|
||||
"""{monitor_id: {latest, recent, uptime, down_since}} for the given monitors.
|
||||
|
||||
`recent` is the last HEARTBEAT_COUNT results oldest-first; `uptime` is
|
||||
{"24h","7d","30d"} percentages; `down_since` is set only for a monitor
|
||||
whose latest check failed.
|
||||
"""
|
||||
ids = [m.id for m in monitors]
|
||||
recent = await _last_n_results(db, ids)
|
||||
uptime = await _uptime_by_monitor(db, ids)
|
||||
out: dict[str, dict] = {}
|
||||
for m in monitors:
|
||||
rows = recent.get(m.id, [])
|
||||
latest = rows[-1] if rows else None
|
||||
out[m.id] = {
|
||||
"latest": latest,
|
||||
"recent": rows,
|
||||
"uptime": uptime.get(m.id, {}),
|
||||
"down_since": None,
|
||||
}
|
||||
if latest is not None and not latest.is_up:
|
||||
out[m.id]["down_since"] = await down_since(db, m.id)
|
||||
return out
|
||||
|
||||
|
||||
async def recent_failures(db, monitor_id: str, limit: int = 10) -> list[MonitorResult]:
|
||||
"""The monitor's latest failed checks, newest first."""
|
||||
return list((await db.execute(
|
||||
select(MonitorResult)
|
||||
.where(MonitorResult.monitor_id == monitor_id, MonitorResult.is_up.is_(False))
|
||||
.order_by(MonitorResult.checked_at.desc())
|
||||
.limit(limit)
|
||||
)).scalars())
|
||||
@@ -8,8 +8,7 @@ from steward.auth.middleware import require_role
|
||||
from steward.core.audit import log_audit
|
||||
from steward.models.users import UserRole
|
||||
from steward.core.settings import (
|
||||
get_all_settings, set_setting, delete_setting,
|
||||
get_stored_plugin_names, find_orphaned_plugins,
|
||||
get_all_settings, set_setting,
|
||||
to_smtp_cfg, to_webhook_cfg, to_ansible_cfg, to_plugins_cfg,
|
||||
to_oidc_cfg, to_ldap_cfg, to_thresholds_cfg,
|
||||
)
|
||||
@@ -126,8 +125,6 @@ _THRESHOLD_FIELDS = [
|
||||
("temp_warn", "thresholds.temp_warn", False), ("temp_crit", "thresholds.temp_crit", False),
|
||||
("uptime_warn", "thresholds.uptime_warn", True), ("uptime_crit", "thresholds.uptime_crit", True),
|
||||
("latency_warn", "ping.threshold.good_ms", False), ("latency_crit", "ping.threshold.warn_ms", False),
|
||||
("tls_warn", "thresholds.tls_warn_days", False), ("tls_crit", "thresholds.tls_crit_days", False),
|
||||
("crash_loop_deaths", "thresholds.crash_loop_deaths", False),
|
||||
]
|
||||
|
||||
# Docker time-series retention windows (days). Read fresh by the cleanup task,
|
||||
@@ -136,7 +133,6 @@ _RETENTION_FIELDS = [
|
||||
("docker_metrics_raw_days", "docker.retention.metrics_raw_days"),
|
||||
("docker_metrics_rollup_days", "docker.retention.metrics_rollup_days"),
|
||||
("docker_events_days", "docker.retention.events_days"),
|
||||
("docker_logs_retention_days", "docker.logs.retention_days"),
|
||||
("metrics_raw_days", "metrics.retention.raw_days"),
|
||||
("metrics_rollup_days", "metrics.retention.rollup_days"),
|
||||
]
|
||||
@@ -174,20 +170,6 @@ async def save_thresholds():
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
await set_setting(db, key, val)
|
||||
# Container-log controls (m79). Checkbox: present ⇒ on (this is a
|
||||
# full-page form, so absence is a genuine "off"). Exclude: comma-split
|
||||
# names. Size: entered in MB, stored as bytes.
|
||||
await set_setting(db, "docker.logs.enabled", "docker_logs_enabled" in form)
|
||||
exclude = [n.strip() for n in form.get("docker_logs_exclude", "").split(",")
|
||||
if n.strip()]
|
||||
await set_setting(db, "docker.logs.exclude", exclude)
|
||||
max_mb = form.get("docker_logs_max_mb", "")
|
||||
if max_mb != "":
|
||||
try:
|
||||
await set_setting(db, "docker.logs.max_bytes_per_container",
|
||||
max(1, int(max_mb)) * 1_000_000)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
await _reload_app_config()
|
||||
await log_audit(current_app, session.get("user_id"), session.get("username", ""),
|
||||
"settings.saved", detail={"section": "thresholds"})
|
||||
@@ -590,7 +572,7 @@ def _build_plugin_cfg_from_form(plugin: dict, form) -> dict:
|
||||
# of a host), not discrete vendor integrations. Presentation-only split — they
|
||||
# still load through the normal plugin mechanism. A plugin.yaml may set
|
||||
# kind: capability|integration to override.
|
||||
CAPABILITY_PLUGINS = {"host_agent", "snmp", "docker"}
|
||||
CAPABILITY_PLUGINS = {"host_agent", "http", "snmp", "docker"}
|
||||
|
||||
|
||||
@settings_bp.get("/plugins/")
|
||||
@@ -598,65 +580,22 @@ CAPABILITY_PLUGINS = {"host_agent", "snmp", "docker"}
|
||||
async def plugins():
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
settings = await get_all_settings(db)
|
||||
stored_plugin_names = await get_stored_plugin_names(db)
|
||||
discovered = _discover_plugins()
|
||||
_merge_plugin_config(discovered, to_plugins_cfg(settings))
|
||||
for p in discovered:
|
||||
p["_kind"] = p.get("kind") or (
|
||||
"capability" if p["_dir"] in CAPABILITY_PLUGINS else "integration")
|
||||
repos = _get_plugin_repos(settings)
|
||||
# Config left behind by plugins that are no longer installed. Surfaced for an
|
||||
# explicit operator decision rather than cleaned up automatically: an external
|
||||
# plugin can be missing because /data/plugins isn't mounted or an install
|
||||
# failed, and silently dropping its row would destroy stored credentials.
|
||||
#
|
||||
# Load failures are folded in so every plugin counted by the admin banner has
|
||||
# a row here. A plugin that failed BECAUSE it has no directory is not in
|
||||
# `discovered`, so without this it would be counted in the banner and shown
|
||||
# nowhere on the page that banner links to.
|
||||
from steward.core.plugin_manager import get_plugin_failures
|
||||
orphans = find_orphaned_plugins(
|
||||
stored_plugin_names,
|
||||
[p["_dir"] for p in discovered],
|
||||
get_plugin_failures(),
|
||||
)
|
||||
return await render_template(
|
||||
"settings/plugins.html",
|
||||
capabilities=[p for p in discovered if p["_kind"] == "capability"],
|
||||
integrations=[p for p in discovered if p["_kind"] != "capability"],
|
||||
discovered_plugins=discovered,
|
||||
orphaned_plugins=orphans,
|
||||
repos=repos,
|
||||
settings=settings,
|
||||
)
|
||||
|
||||
|
||||
@settings_bp.post("/plugins/orphans/<name>/remove/")
|
||||
@require_role(UserRole.admin)
|
||||
async def plugin_orphan_remove(name: str):
|
||||
"""Delete the stored settings row for a plugin that is no longer installed.
|
||||
|
||||
Refuses if the plugin IS installed — that would silently wipe a live
|
||||
plugin's config, and disabling it is what the operator wants there instead.
|
||||
"""
|
||||
installed = {p["_dir"] for p in _discover_plugins()}
|
||||
if name in installed:
|
||||
return redirect(url_for("settings.plugins"))
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
removed = await delete_setting(db, f"plugin.{name}")
|
||||
|
||||
if removed:
|
||||
await _reload_app_config()
|
||||
from steward.core.plugin_index import clear_catalog_cache
|
||||
clear_catalog_cache()
|
||||
await log_audit(
|
||||
current_app, session.get("user_id"), session.get("username", ""),
|
||||
"plugin.settings_removed", entity_type="plugin", entity_id=name)
|
||||
return redirect(url_for("settings.plugins"))
|
||||
|
||||
|
||||
# ── Per-plugin detail (settings) ──────────────────────────────────────────────
|
||||
|
||||
@settings_bp.get("/plugins/<name>/")
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
{#- How to point an AI client at /mcp. `mcp_url` is the endpoint; `token` is the
|
||||
raw token on the page that just created one, else a placeholder. -#}
|
||||
{% set tok = token or "stw_…your token…" %}
|
||||
{% set pre = "background:var(--bg);padding:0.75rem;border-radius:4px;overflow-x:auto;font-size:0.78rem;color:var(--text);margin:0.5rem 0 0;white-space:pre;" %}
|
||||
<div class="card" id="connect">
|
||||
<div class="section-title" style="margin-bottom:0.5rem;">Connect a client</div>
|
||||
<div style="font-size:0.85rem;color:var(--text-muted);line-height:1.55;margin-bottom:0.75rem;">
|
||||
Steward's MCP endpoint. Clients send the token as
|
||||
<code>Authorization: Bearer …</code> on every request.
|
||||
</div>
|
||||
<div style="display:flex;gap:0.5rem;align-items:center;margin-bottom:1rem;">
|
||||
<input type="text" id="mcp-url" value="{{ mcp_url }}" readonly aria-label="MCP endpoint URL"
|
||||
style="flex:1;font-family:ui-monospace,monospace;font-size:0.8rem;background:var(--bg);color:var(--text);">
|
||||
<button type="button" class="btn btn-ghost btn-sm" style="white-space:nowrap;"
|
||||
onclick="navigator.clipboard.writeText(document.getElementById('mcp-url').value).then(()=>this.textContent='Copied')">Copy</button>
|
||||
</div>
|
||||
|
||||
<details open>
|
||||
<summary style="cursor:pointer;font-weight:600;">Claude Code</summary>
|
||||
<pre style="{{ pre }}">claude mcp add --transport http steward {{ mcp_url }} \
|
||||
--header "Authorization: Bearer {{ tok }}"</pre>
|
||||
</details>
|
||||
|
||||
<details style="margin-top:0.75rem;">
|
||||
<summary style="cursor:pointer;font-weight:600;">Claude Desktop</summary>
|
||||
<div style="font-size:0.8rem;color:var(--text-muted);margin-top:0.4rem;line-height:1.5;">
|
||||
Add to <code>claude_desktop_config.json</code>. It bridges through
|
||||
<code>mcp-remote</code>, which needs Node.js.
|
||||
</div>
|
||||
<pre style="{{ pre }}">{
|
||||
"mcpServers": {
|
||||
"steward": {
|
||||
"command": "npx",
|
||||
"args": ["mcp-remote", "{{ mcp_url }}",
|
||||
"--header", "Authorization:${STEWARD_AUTH}"],
|
||||
"env": { "STEWARD_AUTH": "Bearer {{ tok }}" }
|
||||
}
|
||||
}
|
||||
}</pre>
|
||||
</details>
|
||||
|
||||
<details style="margin-top:0.75rem;">
|
||||
<summary style="cursor:pointer;font-weight:600;">Open WebUI and other clients</summary>
|
||||
<div style="font-size:0.8rem;color:var(--text-muted);margin-top:0.4rem;line-height:1.5;">
|
||||
Add an MCP server using the <strong>Streamable HTTP</strong> transport, with the URL
|
||||
above and bearer authentication set to the token. In Open WebUI this is an external
|
||||
tool connection of type MCP; menu names vary between versions.
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:1rem;line-height:1.5;">
|
||||
Behind a reverse proxy, the URL above comes from <strong>Settings → General → Public
|
||||
base URL</strong> when it is set. The proxy must pass the <code>Authorization</code>
|
||||
header through and not buffer responses, since replies stream.
|
||||
</div>
|
||||
</div>
|
||||
@@ -1,40 +0,0 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "_macros.html" import crumbs %}
|
||||
{% block title %}Token created — Steward{% endblock %}
|
||||
{% block breadcrumb %}{{ crumbs([("AI access", "/ai-access/"), ("Token created", "")]) }}{% endblock %}
|
||||
{% block content %}
|
||||
<div style="max-width:640px;">
|
||||
<h1 class="page-title">Token created</h1>
|
||||
|
||||
<div class="card" style="border-color:var(--gold);margin-bottom:1rem;">
|
||||
<div style="font-size:0.8rem;font-weight:600;color:var(--gold);text-transform:uppercase;letter-spacing:0.06em;margin-bottom:0.75rem;">
|
||||
Copy this token now. It can't be shown again.
|
||||
</div>
|
||||
|
||||
<div style="display:flex;gap:0.5rem;align-items:center;">
|
||||
<input type="text" id="api-token" value="{{ raw_token }}" readonly
|
||||
aria-label="API token"
|
||||
style="flex:1;font-family:ui-monospace,monospace;font-size:0.8rem;background:var(--bg);color:var(--text);">
|
||||
<button type="button"
|
||||
onclick="navigator.clipboard.writeText(document.getElementById('api-token').value).then(()=>this.textContent='Copied')"
|
||||
class="btn btn-ghost btn-sm" style="white-space:nowrap;">Copy</button>
|
||||
</div>
|
||||
|
||||
<div style="margin-top:0.75rem;font-size:0.82rem;color:var(--text-muted);">
|
||||
<strong>Name:</strong> {{ token.name }}
|
||||
· <strong>Role:</strong> {{ token.role.value }}
|
||||
· <strong>Expires:</strong>
|
||||
{% if token.expires_at %}{{ token.expires_at.strftime("%Y-%m-%d") }}{% else %}Never{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p style="font-size:0.85rem;color:var(--text-muted);line-height:1.55;">
|
||||
Treat it like a password: anyone holding it can act as you, up to this token's role.
|
||||
If it leaks, revoke it from the AI access page.
|
||||
</p>
|
||||
|
||||
{% with token = raw_token %}{% include "ai_access/_connect.html" %}{% endwith %}
|
||||
|
||||
<a href="/ai-access/" class="btn btn-ghost">Back to AI access</a>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -1,124 +0,0 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}AI access — Steward{% endblock %}
|
||||
{% block head %}
|
||||
<style>
|
||||
.ai-access-grid { display:grid; grid-template-columns:minmax(0,1fr) 320px; gap:1.5rem; align-items:start; }
|
||||
.ai-access-grid .card-flush { overflow-x:auto; }
|
||||
@media (max-width: 900px) { .ai-access-grid { grid-template-columns:minmax(0,1fr); } }
|
||||
</style>
|
||||
{% endblock %}
|
||||
{% block content %}
|
||||
{% set role_help = {
|
||||
"viewer": "Reads everything you can see. Changes nothing.",
|
||||
"operator": "Also runs playbooks and changes monitors and alerts.",
|
||||
"admin": "Full access, the same as an admin in the UI.",
|
||||
} %}
|
||||
<h1 class="page-title">AI access</h1>
|
||||
<p style="color:var(--text-muted);font-size:0.9rem;max-width:720px;margin:-0.5rem 0 1.5rem;line-height:1.55;">
|
||||
A token lets an AI client such as Claude Code connect to Steward over MCP and act as
|
||||
<strong style="color:var(--text);">you</strong>, limited to the role you give the token.
|
||||
Everything a token changes is written to the audit log under your name.
|
||||
</p>
|
||||
|
||||
<div class="ai-access-grid">
|
||||
|
||||
<div>
|
||||
<div class="section-title" style="margin-bottom:0.75rem;">Your tokens</div>
|
||||
{% if rows %}
|
||||
<div class="card-flush">
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Role</th>
|
||||
<th>Created</th>
|
||||
<th>Last used</th>
|
||||
<th>Expires</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for row in rows %}
|
||||
{% set t = row.token %}
|
||||
<tr {% if row.state != 'active' %}style="opacity:0.55;"{% endif %}>
|
||||
<td>
|
||||
<div style="font-weight:600;">{{ t.name }}</div>
|
||||
<div style="font-family:ui-monospace,monospace;font-size:0.75rem;color:var(--text-muted);">{{ t.prefix }}…</div>
|
||||
</td>
|
||||
<td><span class="badge badge-dim">{{ t.role.value }}</span></td>
|
||||
<td style="white-space:nowrap;">{{ t.created_at.strftime("%Y-%m-%d") }}</td>
|
||||
<td style="white-space:nowrap;">
|
||||
{% if t.last_used_at %}{{ t.last_used_at.strftime("%Y-%m-%d %H:%M") }} UTC{% else %}<span style="color:var(--text-muted);">Never</span>{% endif %}
|
||||
</td>
|
||||
<td style="white-space:nowrap;">
|
||||
{% if row.state == 'revoked' %}<span class="badge badge-red">Revoked</span>
|
||||
{% elif row.state == 'expired' %}<span class="badge badge-red">Expired</span>
|
||||
{% elif t.expires_at %}{{ t.expires_at.strftime("%Y-%m-%d") }}
|
||||
{% else %}<span style="color:var(--text-muted);">Never</span>{% endif %}
|
||||
</td>
|
||||
<td class="td-actions">
|
||||
{% if row.state == 'active' %}
|
||||
<form method="post" action="/ai-access/tokens/{{ t.id }}/revoke" style="margin:0;"
|
||||
onsubmit='return confirm({{ ("Revoke “" ~ t.name ~ "”? Any client using it stops working immediately.")|tojson }})'>
|
||||
<button type="submit" class="btn btn-danger btn-sm">Revoke</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="card empty">
|
||||
You have no tokens yet. Create one on the right, then paste it into your AI client.
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<div class="section-title" style="margin-bottom:0.75rem;">Create a token</div>
|
||||
<div class="card" style="margin-bottom:0;">
|
||||
<form method="post" action="/ai-access/tokens">
|
||||
<div class="form-group">
|
||||
<label for="tok-name">Name</label>
|
||||
<input type="text" id="tok-name" name="name" required maxlength="128"
|
||||
placeholder="e.g. Claude Code on my laptop">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="tok-role">Role</label>
|
||||
<select id="tok-role" name="role">
|
||||
{% for r in grantable %}
|
||||
<option value="{{ r.value }}" {% if loop.first %}selected{% endif %}>{{ r.value|capitalize }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.4rem;line-height:1.5;">
|
||||
{% for r in grantable %}
|
||||
<div><strong style="color:var(--text);">{{ r.value|capitalize }}:</strong> {{ role_help[r.value] }}</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="tok-exp">Expires after</label>
|
||||
<select id="tok-exp" name="expires_in_days">
|
||||
{% for value, label in expiry_choices %}
|
||||
<option value="{{ value }}">{{ label }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
</div>
|
||||
<button type="submit" class="btn">Create token</button>
|
||||
</form>
|
||||
</div>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.75rem;line-height:1.5;">
|
||||
The token is shown <strong>once</strong>. A token never acts above your own role
|
||||
({{ user_role.value }}), even if your role is lowered later.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div style="max-width:720px;margin-top:1.5rem;">
|
||||
{% set token = none %}
|
||||
{% include "ai_access/_connect.html" %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -20,7 +20,7 @@
|
||||
<div style="display:grid;grid-template-columns:auto 1fr;gap:0.4rem 1rem;font-size:0.9rem;margin-bottom:1rem;">
|
||||
<span style="color:var(--text-muted);">ID</span><span style="color:var(--text-dim);font-family:monospace;">{{ run.id }}</span>
|
||||
<span style="color:var(--text-muted);">Playbook</span><span>{{ run.playbook_path }}</span>
|
||||
<span style="color:var(--text-muted);">Scope</span><span title="{{ run.inventory_scope }}">{{ scope_label or run.inventory_scope }}</span>
|
||||
<span style="color:var(--text-muted);">Scope</span><span style="font-family:ui-monospace,monospace;font-size:0.85rem;">{{ run.inventory_scope }}</span>
|
||||
<span style="color:var(--text-muted);">Source</span><span>{{ run.source_name }}</span>
|
||||
<span style="color:var(--text-muted);">Triggered by</span><span>{{ triggered_label }}</span>
|
||||
<span style="color:var(--text-muted);">Status</span><span style="color:{{ status_color }};font-weight:bold;">{{ run.status.value.upper() }}</span>
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
<tr style="{% if not s.enabled %}opacity:0.55;{% endif %}">
|
||||
<td style="font-weight:500;">{{ s.name }}</td>
|
||||
<td style="font-size:0.85rem;"><span style="color:var(--text-dim);">{{ s.source_name }}</span> / {{ s.playbook_path }}</td>
|
||||
<td style="font-size:0.85rem;" title="{{ s.inventory_scope }}">{{ row.scope_label }}</td>
|
||||
<td style="font-size:0.82rem;color:var(--text-muted);font-family:ui-monospace,monospace;">{{ s.inventory_scope }}</td>
|
||||
<td style="font-size:0.85rem;">{{ fmt_interval(s.interval_seconds) }}</td>
|
||||
<td style="font-size:0.82rem;">
|
||||
{% if s.last_error %}
|
||||
|
||||
@@ -313,7 +313,6 @@ body.dash-editing .widget-drawer { transform:translateY(0); }
|
||||
<div class="nav-group">
|
||||
<div class="nav-group-label">Automation</div>
|
||||
<a href="/ansible/" {% if _p.startswith('/ansible') %}class="active" aria-current="page"{% endif %}>Ansible</a>
|
||||
<a href="/ai-access/" {% if _p.startswith('/ai-access') %}class="active" aria-current="page"{% endif %}>AI access</a>
|
||||
</div>
|
||||
{% if session.user_role == 'admin' %}
|
||||
<div class="nav-group">
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Database unavailable — Steward{% endblock %}
|
||||
{% block content %}
|
||||
<div style="text-align:center; padding: 4rem 2rem; font-family: var(--font-serif);">
|
||||
<h1 style="color: var(--yellow); font-size: 2.5rem; margin-bottom: 0.5rem;">503</h1>
|
||||
<p style="color: var(--text); font-size: 1.2rem; margin-bottom: 0.25rem;">The records are out of reach.</p>
|
||||
<p style="color: var(--text-dim); margin-bottom: 0.5rem; max-width: 34rem; margin-left:auto; margin-right:auto;">
|
||||
Steward is running, but cannot reach its database. Nothing has been lost —
|
||||
this page will work again as soon as the database is back.
|
||||
</p>
|
||||
<p style="color: var(--text-dim); font-size: 0.85rem; margin-bottom: 1.5rem;">
|
||||
Steward already retried for a moment before showing this.
|
||||
</p>
|
||||
<a href="{{ request.path }}" class="btn">Try again</a>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -60,7 +60,7 @@
|
||||
</form>
|
||||
</div>
|
||||
<p style="color:var(--text-muted);font-size:0.82rem;margin:0 0 0.75rem;max-width:720px;">
|
||||
Built-in ways to monitor your hosts (agent metrics, SNMP, Docker). These are
|
||||
Built-in ways to monitor your hosts (agent metrics, HTTP/uptime, SNMP, Docker). These are
|
||||
facets of a host — you'll see their data in the <a href="/hosts/">Hosts</a> and
|
||||
<a href="/status">Status</a> sections, not as separate areas. On by default.
|
||||
</p>
|
||||
@@ -88,63 +88,6 @@
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{# ── Orphaned settings ─────────────────────────────────────────────────────── #}
|
||||
{# Hidden entirely when there is nothing to clean up — an empty "Configured but
|
||||
not installed" panel would read as a problem rather than a clean state. #}
|
||||
{% if orphaned_plugins %}
|
||||
<div style="margin-bottom:1.5rem;max-width:720px;">
|
||||
<div class="section-title" style="margin-bottom:0.4rem;">Configured but not installed</div>
|
||||
<p style="color:var(--text-muted);font-size:0.82rem;margin:0 0 0.75rem;">
|
||||
Plugins Steward has settings for, or tried to load, but cannot find on disk — including
|
||||
anything counted by the "failed to load" banner. This usually means the plugin was
|
||||
removed, but it also happens when an external plugin directory isn't mounted or an
|
||||
install failed part-way, so nothing is deleted automatically. Removing an entry
|
||||
discards that plugin's saved configuration, including any credentials.
|
||||
</p>
|
||||
<div style="display:grid;gap:0.6rem;">
|
||||
{% for orphan in orphaned_plugins %}
|
||||
<div class="card" style="padding:0.85rem 1rem;display:flex;align-items:center;gap:0.75rem;">
|
||||
<span style="width:8px;height:8px;border-radius:50%;flex-shrink:0;
|
||||
background:{{ 'var(--red)' if orphan.reason else 'var(--yellow)' }};"
|
||||
title="{{ orphan.reason or 'Configured but not installed' }}"></span>
|
||||
<div style="flex:1;min-width:0;">
|
||||
<div style="display:flex;align-items:baseline;gap:0.5rem;flex-wrap:wrap;">
|
||||
<span style="font-weight:600;font-size:0.9rem;color:var(--text);">{{ orphan.name }}</span>
|
||||
{% if orphan.reason %}
|
||||
<span style="font-size:0.72rem;padding:0.1em 0.45em;border-radius:3px;
|
||||
background:color-mix(in srgb,var(--red) 15%,var(--bg));color:var(--red);">Failed to load</span>
|
||||
{% else %}
|
||||
<span style="font-size:0.72rem;padding:0.1em 0.45em;border-radius:3px;
|
||||
background:var(--yellow-dim);color:var(--yellow);">Not installed</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% if orphan.reason %}
|
||||
<div style="font-size:0.77rem;color:var(--red);margin-top:0.1rem;
|
||||
white-space:nowrap;overflow:hidden;text-overflow:ellipsis;"
|
||||
title="{{ orphan.reason }}">{{ orphan.reason }}</div>
|
||||
{% endif %}
|
||||
<div style="font-size:0.8rem;color:var(--text-muted);margin-top:0.1rem;">
|
||||
{% if orphan.removable %}
|
||||
Settings key <code>plugin.{{ orphan.name }}</code> has no matching plugin.
|
||||
{% else %}
|
||||
Enabled by a built-in default with no stored settings to remove — this is a
|
||||
packaging bug, not leftover configuration. Please report it.
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
{% if orphan.removable %}
|
||||
<form method="post" action="/settings/plugins/orphans/{{ orphan.name }}/remove/"
|
||||
onsubmit="return confirm('Remove stored settings for "{{ orphan.name }}"? Its saved configuration, including any credentials, will be discarded. This cannot be undone.');"
|
||||
style="margin:0;flex-shrink:0;">
|
||||
<button type="submit" class="btn btn-danger btn-sm" style="font-size:0.78rem;">Remove</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{# ── Plugin Repositories ───────────────────────────────────────────────────── #}
|
||||
<div style="margin-bottom:2rem;max-width:720px;">
|
||||
<div class="section-title" style="margin-bottom:0.75rem;">Plugin Repositories</div>
|
||||
|
||||
@@ -40,18 +40,6 @@
|
||||
{{ pair("Temperature", "temp_warn", "thresholds.temp_warn", "temp_crit", "thresholds.temp_crit", "°C", "Hottest reported sensor per host.") }}
|
||||
{{ pair("Ping latency", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "Round-trip time. Shared with the Ping widget's good/warn colouring.") }}
|
||||
{{ pair("Uptime / SLA", "uptime_warn", "thresholds.uptime_warn", "uptime_crit", "thresholds.uptime_crit", "%", "Lower is worse — amber below warn, red below crit.", step="0.1") }}
|
||||
{{ pair("TLS certificate expiry", "tls_warn", "thresholds.tls_warn_days", "tls_crit", "thresholds.tls_crit_days", "days left", "HTTPS monitors whose certificate expires within these many days are flagged. Lower is worse.") }}
|
||||
|
||||
<div class="form-group" style="margin-bottom:0.25rem;">
|
||||
<label>Container crash loop <span style="color:var(--text-muted);font-size:0.8rem;">(deaths per hour)</span></label>
|
||||
<div style="margin-top:0.25rem;">
|
||||
<input type="number" name="crash_loop_deaths" min="1" step="1"
|
||||
value="{{ settings['thresholds.crash_loop_deaths'] }}" style="max-width:110px;">
|
||||
</div>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">
|
||||
A container that dies (or is OOM-killed) this many times within an hour is reported as crash-looping.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% macro days(label, field, key, hint) %}
|
||||
@@ -82,54 +70,6 @@
|
||||
"Keep container start/stop/die/health history this long.") }}
|
||||
</div>
|
||||
|
||||
<div class="card" style="max-width:640px;margin-top:1rem;">
|
||||
<h2 class="section-title" style="margin-bottom:0.5rem;">Container logs</h2>
|
||||
<p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">
|
||||
The host agent tails each container's logs and pushes them here, viewable per
|
||||
container. On by default for every container; storage is bounded by a
|
||||
per-container ring (oldest lines rotate out once the age or size cap is hit,
|
||||
whichever comes first). Applied by the hourly cleanup and on ingest.
|
||||
</p>
|
||||
|
||||
<div class="form-group" style="margin-bottom:1.1rem;">
|
||||
<label style="display:flex;align-items:center;gap:0.5rem;cursor:pointer;">
|
||||
<input type="checkbox" name="docker_logs_enabled" style="width:auto;"
|
||||
{% if settings["docker.logs.enabled"] %}checked{% endif %}>
|
||||
Collect container logs
|
||||
</label>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">
|
||||
Global kill-switch. When off, pushed log lines are dropped and no new logs are stored.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="form-group" style="margin-bottom:1.1rem;">
|
||||
<label>Exclude containers</label>
|
||||
<div style="margin-top:0.25rem;">
|
||||
<input type="text" name="docker_logs_exclude"
|
||||
value="{{ settings['docker.logs.exclude'] | join(', ') }}"
|
||||
placeholder="watchtower, some-chatty-service" style="width:100%;max-width:420px;">
|
||||
</div>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">
|
||||
Comma-separated container names whose logs are dropped on ingest (e.g. known-noisy ones).
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{ days("Log retention", "docker_logs_retention_days", "docker.logs.retention_days",
|
||||
"Keep each container's log lines at most this long before rotating them out.") }}
|
||||
|
||||
<div class="form-group" style="margin-bottom:0.25rem;">
|
||||
<label>Max size per container <span style="color:var(--text-muted);font-size:0.8rem;">(MB)</span></label>
|
||||
<div style="margin-top:0.25rem;">
|
||||
<input type="number" name="docker_logs_max_mb" min="1" step="1"
|
||||
value="{{ (settings['docker.logs.max_bytes_per_container'] // 1000000) or 1 }}"
|
||||
style="max-width:110px;">
|
||||
</div>
|
||||
<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">
|
||||
Newest lines are kept up to this size per container; older lines rotate out first.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card" style="max-width:640px;margin-top:1rem;">
|
||||
<h2 class="section-title" style="margin-bottom:0.5rem;">Host metrics retention</h2>
|
||||
<p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">
|
||||
|
||||
@@ -1,21 +1,11 @@
|
||||
"""The bundled first-party playbook source is always present and discoverable."""
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from steward.ansible.sources import (
|
||||
BUILTIN_SOURCE_NAME,
|
||||
discover_playbook_meta,
|
||||
discover_playbooks,
|
||||
get_sources,
|
||||
)
|
||||
|
||||
|
||||
def _bundled_content(rel_path: str) -> str:
|
||||
builtin = get_sources({"sources": []})[0]
|
||||
return (Path(builtin["path"]) / rel_path).read_text()
|
||||
|
||||
|
||||
def test_builtin_source_is_first_and_local():
|
||||
sources = get_sources({"sources": []})
|
||||
assert sources[0]["name"] == BUILTIN_SOURCE_NAME
|
||||
@@ -27,33 +17,3 @@ def test_bundled_playbooks_are_discoverable():
|
||||
playbooks = discover_playbooks(builtin["path"])
|
||||
assert "maintenance/docker_prune.yml" in playbooks
|
||||
assert "host_agent/install.yml" in playbooks
|
||||
|
||||
|
||||
def test_docker_prune_playbook_parses_and_keeps_meta():
|
||||
"""The prune playbook stays valid YAML and keeps its self-describing meta
|
||||
(a destructive maintenance run that must prompt for confirmation)."""
|
||||
content = _bundled_content("maintenance/docker_prune.yml")
|
||||
plays = yaml.safe_load(content)
|
||||
assert isinstance(plays, list) and plays # at least one play
|
||||
|
||||
meta = discover_playbook_meta(content)
|
||||
assert meta["confirm"] is True
|
||||
assert meta["category"] == "maintenance"
|
||||
assert meta["description"]
|
||||
|
||||
|
||||
def test_docker_prune_supports_all_three_targets():
|
||||
"""M78 drives the three prune buttons via a single `prune_target` var;
|
||||
`system` is the default so pre-M78 callers (no var set) are unchanged."""
|
||||
content = _bundled_content("maintenance/docker_prune.yml")
|
||||
play = yaml.safe_load(content)[0]
|
||||
assert play["vars"]["prune_target"] == "system"
|
||||
|
||||
# Every target the routes/UI can send has a matching guarded task.
|
||||
guards = {
|
||||
task.get("when")
|
||||
for task in play["tasks"]
|
||||
if isinstance(task.get("when"), str) and "prune_target" in task["when"]
|
||||
}
|
||||
for target in ("containers", "images", "system"):
|
||||
assert f"prune_target == '{target}'" in guards
|
||||
|
||||
@@ -1,307 +0,0 @@
|
||||
"""DB connection resilience: pool, startup readiness, and request-time outage.
|
||||
|
||||
Covers the three recovery gaps that let a database restart take the app down:
|
||||
the engine handing out stale pooled connections (#2626), startup assuming the
|
||||
DB is reachable and ready the instant it asks (#2627), and a request finding
|
||||
the database gone and failing with an opaque 500 (#2635).
|
||||
"""
|
||||
import types
|
||||
import pytest
|
||||
|
||||
from steward import database
|
||||
from steward.database import (
|
||||
POOL_RECYCLE_SECONDS,
|
||||
_describe,
|
||||
_fatal_sqlstate,
|
||||
init_db,
|
||||
wait_for_database,
|
||||
)
|
||||
|
||||
|
||||
class _PGError(Exception):
|
||||
"""Stand-in for an asyncpg error, which carries a SQLSTATE attribute."""
|
||||
|
||||
def __init__(self, message: str, sqlstate: str | None = None):
|
||||
super().__init__(message)
|
||||
self.sqlstate = sqlstate
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def no_sleep(monkeypatch):
|
||||
"""Drive the retry budget off a fake clock instead of wall time.
|
||||
|
||||
sleep() records its duration and advances the clock by exactly that much,
|
||||
so backoff and timeout behaviour are deterministic rather than dependent on
|
||||
how fast the test machine happens to run.
|
||||
"""
|
||||
slept: list[float] = []
|
||||
clock = {"now": 0.0}
|
||||
|
||||
def fake_sleep(seconds: float) -> None:
|
||||
slept.append(seconds)
|
||||
clock["now"] += seconds
|
||||
|
||||
monkeypatch.setattr(database.time, "sleep", fake_sleep)
|
||||
monkeypatch.setattr(database.time, "monotonic", lambda: clock["now"])
|
||||
return slept
|
||||
|
||||
|
||||
def _probe_raising(*errors, then_succeed: bool = True):
|
||||
"""Async probe stub raising the given errors in order.
|
||||
|
||||
Once the list is exhausted it succeeds, unless then_succeed is False — in
|
||||
which case it keeps raising the last error forever (a DB that never comes
|
||||
back).
|
||||
"""
|
||||
calls = {"n": 0}
|
||||
|
||||
async def probe(db_url):
|
||||
i = calls["n"]
|
||||
calls["n"] += 1
|
||||
if i < len(errors):
|
||||
raise errors[i]
|
||||
if not then_succeed:
|
||||
raise errors[-1]
|
||||
|
||||
probe.calls = calls
|
||||
return probe
|
||||
|
||||
|
||||
# ── engine pool configuration (issue #2626) ──────────────────────────────────
|
||||
|
||||
|
||||
def test_engine_enables_pre_ping_and_recycle():
|
||||
app = types.SimpleNamespace(
|
||||
config={"DATABASE_URL": "postgresql+asyncpg://u:p@localhost/db"}
|
||||
)
|
||||
init_db(app)
|
||||
|
||||
# _pre_ping / _recycle are SQLAlchemy pool internals; there is no public
|
||||
# accessor, and these are exactly the settings a DB restart depends on.
|
||||
pool = app._db_engine.sync_engine.pool
|
||||
assert pool._pre_ping is True
|
||||
assert pool._recycle == POOL_RECYCLE_SECONDS
|
||||
|
||||
|
||||
def test_pool_recycle_is_positive():
|
||||
# A non-positive recycle disables age-based retirement entirely.
|
||||
assert POOL_RECYCLE_SECONDS > 0
|
||||
|
||||
|
||||
# ── retry classification ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_fatal_sqlstate_detects_bad_password():
|
||||
exc = _PGError("password authentication failed", sqlstate="28P01")
|
||||
assert _fatal_sqlstate(exc) == "28P01"
|
||||
|
||||
|
||||
def test_fatal_sqlstate_finds_sqlstate_through_cause_chain():
|
||||
inner = _PGError("database does not exist", sqlstate="3D000")
|
||||
outer = RuntimeError("wrapped by sqlalchemy")
|
||||
outer.__cause__ = inner
|
||||
assert _fatal_sqlstate(outer) == "3D000"
|
||||
|
||||
|
||||
def test_transient_errors_are_not_fatal():
|
||||
assert _fatal_sqlstate(OSError(-2, "Name or service not known")) is None
|
||||
# "the database system is not yet accepting connections" — resolves on its own.
|
||||
assert _fatal_sqlstate(_PGError("not yet accepting", sqlstate="57P03")) is None
|
||||
|
||||
|
||||
def test_describe_unwraps_to_innermost_cause():
|
||||
inner = OSError("Name or service not known")
|
||||
outer = RuntimeError("sqlalchemy boilerplate")
|
||||
outer.__cause__ = inner
|
||||
assert "Name or service not known" in _describe(outer)
|
||||
|
||||
|
||||
def test_describe_survives_self_referential_cause():
|
||||
exc = RuntimeError("loop")
|
||||
exc.__cause__ = exc
|
||||
assert "loop" in _describe(exc)
|
||||
|
||||
|
||||
# ── wait_for_database (issue #2627) ──────────────────────────────────────────
|
||||
|
||||
|
||||
def test_returns_immediately_when_db_is_up(monkeypatch, no_sleep):
|
||||
probe = _probe_raising()
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=10)
|
||||
|
||||
assert probe.calls["n"] == 1
|
||||
assert no_sleep == []
|
||||
|
||||
|
||||
def test_retries_dns_failure_then_succeeds(monkeypatch, no_sleep):
|
||||
# The reported failure: gaierror -2 while Docker DNS is not yet serving `db`.
|
||||
probe = _probe_raising(
|
||||
OSError(-2, "Name or service not known"),
|
||||
OSError(-2, "Name or service not known"),
|
||||
)
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=30)
|
||||
|
||||
assert probe.calls["n"] == 3
|
||||
assert len(no_sleep) == 2
|
||||
|
||||
|
||||
def test_retries_while_postgres_is_still_recovering(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(
|
||||
_PGError("the database system is not yet accepting connections", "57P03"),
|
||||
)
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=30)
|
||||
|
||||
assert probe.calls["n"] == 2
|
||||
|
||||
|
||||
def test_backoff_grows_between_attempts(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(*[OSError("refused")] * 4)
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=60)
|
||||
|
||||
assert no_sleep == sorted(no_sleep), "delays should be non-decreasing"
|
||||
assert no_sleep[-1] > no_sleep[0], "backoff should grow, not stay flat"
|
||||
|
||||
|
||||
def test_backoff_is_capped(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(*[OSError("refused")] * 12)
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=600)
|
||||
|
||||
assert max(no_sleep) <= database._BACKOFF_MAX_SECONDS
|
||||
|
||||
|
||||
def test_bad_credentials_fail_immediately_without_retrying(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(_PGError("password authentication failed", "28P01"))
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
with pytest.raises(_PGError):
|
||||
wait_for_database("postgresql+asyncpg://u:bad@db/steward", timeout_seconds=30)
|
||||
|
||||
# Waiting cannot fix a wrong password — don't burn the whole budget on it.
|
||||
assert probe.calls["n"] == 1
|
||||
assert no_sleep == []
|
||||
|
||||
|
||||
def test_missing_database_fails_immediately(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(_PGError("database does not exist", "3D000"))
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
with pytest.raises(_PGError):
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/nope", timeout_seconds=30)
|
||||
|
||||
assert probe.calls["n"] == 1
|
||||
|
||||
|
||||
def test_gives_up_after_budget_with_actionable_message(monkeypatch, no_sleep):
|
||||
probe = _probe_raising(
|
||||
OSError("Name or service not known"), then_succeed=False
|
||||
)
|
||||
monkeypatch.setattr(database, "_probe", probe)
|
||||
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
wait_for_database("postgresql+asyncpg://u:p@db/steward", timeout_seconds=2)
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "did not become available" in message
|
||||
# The operator needs the underlying cause, not just "timed out".
|
||||
assert "Name or service not known" in message
|
||||
|
||||
|
||||
# ── ensure_database_reachable (issue #2635) ──────────────────────────────────
|
||||
|
||||
|
||||
class _FakeEngine:
|
||||
"""Minimal stand-in for AsyncEngine.connect() as an async context manager."""
|
||||
|
||||
def __init__(self, *errors, then_succeed: bool = True):
|
||||
self.errors = list(errors)
|
||||
self.then_succeed = then_succeed
|
||||
self.calls = 0
|
||||
|
||||
def connect(self):
|
||||
engine = self
|
||||
|
||||
class _Ctx:
|
||||
async def __aenter__(self):
|
||||
index = engine.calls
|
||||
engine.calls += 1
|
||||
if index < len(engine.errors):
|
||||
raise engine.errors[index]
|
||||
if not engine.then_succeed:
|
||||
raise engine.errors[-1]
|
||||
return object()
|
||||
|
||||
async def __aexit__(self, *exc_info):
|
||||
return False
|
||||
|
||||
return _Ctx()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def no_async_sleep(monkeypatch):
|
||||
"""Record awaited retry delays without spending them."""
|
||||
slept: list[float] = []
|
||||
|
||||
async def fake_sleep(seconds: float) -> None:
|
||||
slept.append(seconds)
|
||||
|
||||
monkeypatch.setattr(database.asyncio, "sleep", fake_sleep)
|
||||
return slept
|
||||
|
||||
|
||||
async def test_reachable_on_first_try_costs_no_retries(no_async_sleep):
|
||||
engine = _FakeEngine()
|
||||
await database.ensure_database_reachable(engine)
|
||||
assert engine.calls == 1
|
||||
assert no_async_sleep == []
|
||||
|
||||
|
||||
async def test_rides_out_a_brief_outage(no_async_sleep):
|
||||
# The reported shape: DNS gone while the db container restarts.
|
||||
engine = _FakeEngine(OSError(-2, "Name or service not known"))
|
||||
await database.ensure_database_reachable(engine)
|
||||
assert engine.calls == 2
|
||||
assert no_async_sleep == [0.25]
|
||||
|
||||
|
||||
async def test_uses_the_full_retry_schedule_before_giving_up(no_async_sleep):
|
||||
engine = _FakeEngine(OSError("refused"), then_succeed=False)
|
||||
with pytest.raises(database.DatabaseUnavailable):
|
||||
await database.ensure_database_reachable(engine)
|
||||
|
||||
# One attempt per delay, plus a final attempt after the last wait.
|
||||
assert engine.calls == len(database._REQUEST_RETRY_DELAYS) + 1
|
||||
assert no_async_sleep == list(database._REQUEST_RETRY_DELAYS)
|
||||
|
||||
|
||||
async def test_request_budget_stays_short(no_async_sleep):
|
||||
# A person is waiting on this; guard against the schedule growing into a hang.
|
||||
assert sum(database._REQUEST_RETRY_DELAYS) <= 3.0
|
||||
|
||||
|
||||
async def test_unavailable_error_carries_the_underlying_cause(no_async_sleep):
|
||||
engine = _FakeEngine(
|
||||
OSError("Name or service not known"), then_succeed=False)
|
||||
with pytest.raises(database.DatabaseUnavailable) as excinfo:
|
||||
await database.ensure_database_reachable(engine)
|
||||
assert "Name or service not known" in str(excinfo.value)
|
||||
|
||||
|
||||
async def test_bad_credentials_are_not_masked_as_unavailable(no_async_sleep):
|
||||
# Retrying a wrong password would be pointless, and reporting it as
|
||||
# "database unavailable" would send the operator chasing the wrong problem.
|
||||
engine = _FakeEngine(
|
||||
_PGError("password authentication failed", "28P01"), then_succeed=False)
|
||||
with pytest.raises(_PGError):
|
||||
await database.ensure_database_reachable(engine)
|
||||
assert engine.calls == 1
|
||||
assert no_async_sleep == []
|
||||
@@ -128,13 +128,3 @@ def test_yaml_injected_hostvars_survive():
|
||||
def test_yaml_empty_targets():
|
||||
parsed = yaml.safe_load(inventory_to_yaml(generate_inventory([])))
|
||||
assert parsed == {"all": {"hosts": {}}}
|
||||
|
||||
|
||||
def test_describe_scope_names_what_a_run_targeted():
|
||||
from steward.ansible.inventory_gen import describe_scope
|
||||
targets, groups = {"t1": "web-1"}, {"g1": "Docker Swarm"}
|
||||
assert describe_scope("steward:all", targets, groups) == "all inventory targets"
|
||||
assert describe_scope("steward:target:t1", targets, groups) == "target web-1"
|
||||
assert describe_scope("steward:group:g1", targets, groups) == "group Docker Swarm"
|
||||
assert describe_scope("steward:target:abcdef0123", targets, groups) == "target abcdef01 (deleted)"
|
||||
assert describe_scope("repo:homelab:hosts.yml", targets, groups) == "homelab/hosts.yml"
|
||||
|
||||
+12
-53
@@ -8,10 +8,7 @@ import asyncio
|
||||
|
||||
from steward.models.monitors import Monitor, MONITOR_TYPE_VALUES
|
||||
from steward.monitors import runner as runner_mod
|
||||
import pytest
|
||||
|
||||
from steward.monitors.routes import _form_opts
|
||||
from steward.monitors.service import build_config, monitor_fields, normalise_target
|
||||
from steward.monitors.routes import _build_config, _normalise_target
|
||||
|
||||
|
||||
def _monitor(mtype, target="example.com", cfg=None):
|
||||
@@ -88,64 +85,26 @@ def test_run_monitor_dns_passes_expected_ip(monkeypatch):
|
||||
assert res["resolved_ip"] == "192.0.2.1"
|
||||
|
||||
|
||||
# ── config validation (shared by the form and the MCP tools) ─────────────────
|
||||
# ── route form helpers ────────────────────────────────────────────────────────
|
||||
|
||||
def test_normalise_target_adds_https_for_bare_http_host():
|
||||
assert normalise_target("http", "example.com") == "https://example.com"
|
||||
assert normalise_target("http", "http://x.com") == "http://x.com"
|
||||
assert normalise_target("tcp", "example.com") == "example.com"
|
||||
assert _normalise_target("http", "example.com") == "https://example.com"
|
||||
assert _normalise_target("http", "http://x.com") == "http://x.com"
|
||||
assert _normalise_target("tcp", "example.com") == "example.com"
|
||||
|
||||
|
||||
class _Form(dict):
|
||||
"""Minimal stand-in for a Quart form (dict + 'in' membership for checkboxes)."""
|
||||
|
||||
|
||||
def _from_form(form, mtype):
|
||||
return build_config(mtype, _form_opts(form, mtype))
|
||||
def test_build_config_per_type():
|
||||
assert _build_config(_Form(port="8080"), "tcp") == {"port": 8080}
|
||||
assert _build_config(_Form(), "tcp") == {"port": 80}
|
||||
assert _build_config(_Form(expected_ip="10.0.0.1"), "dns") == {"expected_ip": "10.0.0.1"}
|
||||
assert _build_config(_Form(), "dns") == {"expected_ip": None}
|
||||
assert _build_config(_Form(), "icmp") == {}
|
||||
|
||||
|
||||
def test_build_config_from_the_form_per_type():
|
||||
assert _from_form(_Form(port="8080"), "tcp") == {"port": 8080}
|
||||
assert _from_form(_Form(), "tcp") == {"port": 80}
|
||||
assert _from_form(_Form(expected_ip="10.0.0.1"), "dns") == {"expected_ip": "10.0.0.1"}
|
||||
assert _from_form(_Form(), "dns") == {"expected_ip": None}
|
||||
assert _from_form(_Form(), "icmp") == {}
|
||||
|
||||
http = _from_form(_Form(method="get", expected_status="201", follow_redirects="on"), "http")
|
||||
http = _build_config(_Form(method="get", expected_status="201", follow_redirects="on"), "http")
|
||||
assert http["method"] == "GET" and http["expected_status"] == 201
|
||||
assert http["follow_redirects"] is True # present in form → checked
|
||||
assert http["verify_ssl"] is False # absent from form → unchecked
|
||||
|
||||
|
||||
def test_build_config_from_typed_values_defaults_to_safe_checks():
|
||||
http = build_config("http", {"headers": {"X-Key": "v"}})
|
||||
assert http["follow_redirects"] is True and http["verify_ssl"] is True
|
||||
assert http["headers"] == {"X-Key": "v"} and http["timeout_seconds"] == 10
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mtype, opts, why", [
|
||||
("tcp", {"port": 70000}, "port must be between 1 and 65535"),
|
||||
("tcp", {"port": "http"}, "port must be a whole number"),
|
||||
("tcp", {"expected_ip": "1.2.3.4"}, "does not take expected_ip; it takes port"),
|
||||
("icmp", {"port": 22}, "takes no config"),
|
||||
("http", {"method": "DELETE"}, "not one of GET, HEAD, POST"),
|
||||
("http", {"expected_status": 42}, "between 100 and 599"),
|
||||
("http", {"timeout_seconds": 0}, "between 1 and 60"),
|
||||
("http", {"headers": "{not json"}, "JSON object"),
|
||||
("http", {"headers": {"X": 1}}, "name: value strings"),
|
||||
])
|
||||
def test_build_config_refuses_with_a_reason(mtype, opts, why):
|
||||
with pytest.raises(ValueError, match=why):
|
||||
build_config(mtype, opts)
|
||||
|
||||
|
||||
def test_monitor_fields():
|
||||
f = monitor_fields("HTTP", target="nas.lan", opts={"expected_status": 204})
|
||||
assert (f["type"], f["target"], f["name"]) == ("http", "https://nas.lan", "https://nas.lan")
|
||||
assert f["check_interval_seconds"] == 0
|
||||
with pytest.raises(ValueError, match="'smtp' is not one of icmp, tcp, dns, http"):
|
||||
monitor_fields("smtp", target="x")
|
||||
with pytest.raises(ValueError, match="needs a target"):
|
||||
monitor_fields("icmp", target=" ")
|
||||
with pytest.raises(ValueError, match="check_interval_seconds must be at least 0"):
|
||||
monitor_fields("icmp", target="x", check_interval_seconds=-5)
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
"""Guards against plugin settings outliving the plugin they configure.
|
||||
|
||||
The `http` plugin was folded into the unified Monitor entity, but its
|
||||
`plugin.http` DEFAULTS entry survived — so the operator saw a plugin that no
|
||||
longer exists reported as enabled, with no way to clear it (deleting the row
|
||||
did nothing; the default reasserted it). These tests make that class of drift a
|
||||
CI failure instead of a support question.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from steward.core.settings import DEFAULTS, find_orphaned_plugins
|
||||
from steward.settings.routes import CAPABILITY_PLUGINS
|
||||
|
||||
BUNDLED_PLUGINS_DIR = Path(__file__).resolve().parents[2] / "plugins"
|
||||
|
||||
|
||||
def _bundled_plugin_names() -> set[str]:
|
||||
"""Plugin dirs shipped inside the image, identified by their plugin.yaml."""
|
||||
return {
|
||||
entry.name
|
||||
for entry in BUNDLED_PLUGINS_DIR.iterdir()
|
||||
if entry.is_dir() and (entry / "plugin.yaml").exists()
|
||||
}
|
||||
|
||||
|
||||
def _default_plugin_names() -> set[str]:
|
||||
prefix = "plugin."
|
||||
return {k[len(prefix):] for k in DEFAULTS if k.startswith(prefix)}
|
||||
|
||||
|
||||
def test_bundled_plugins_dir_is_discoverable():
|
||||
# Guards the test itself: a wrong path would make everything below vacuous.
|
||||
assert _bundled_plugin_names(), f"no bundled plugins found under {BUNDLED_PLUGINS_DIR}"
|
||||
|
||||
|
||||
def test_every_default_plugin_key_has_a_real_bundled_plugin():
|
||||
"""A plugin.* default naming a non-existent plugin is always a bug.
|
||||
|
||||
Bundled plugins ship in the image and version atomically with core, so
|
||||
unlike external plugins they cannot be transiently missing — there is no
|
||||
benign reason for this to fail.
|
||||
"""
|
||||
missing = _default_plugin_names() - _bundled_plugin_names()
|
||||
assert not missing, (
|
||||
f"DEFAULTS declares plugin(s) with no bundled directory: {sorted(missing)}. "
|
||||
f"If the plugin was removed, delete its plugin.<name> key from DEFAULTS "
|
||||
f"and add a migration dropping the stored row."
|
||||
)
|
||||
|
||||
|
||||
def test_http_plugin_default_is_gone():
|
||||
# Explicit regression: this exact key is what the operator hit.
|
||||
assert "plugin.http" not in DEFAULTS
|
||||
|
||||
|
||||
def test_capability_plugins_all_exist():
|
||||
"""CAPABILITY_PLUGINS classifies discovered plugins; stale names are dead weight."""
|
||||
missing = CAPABILITY_PLUGINS - _bundled_plugin_names()
|
||||
assert not missing, f"CAPABILITY_PLUGINS names non-existent plugin(s): {sorted(missing)}"
|
||||
|
||||
|
||||
# ── orphan detection ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_orphans_are_stored_names_with_no_installed_plugin():
|
||||
rows = find_orphaned_plugins(["docker", "traefik", "ancient"], ["docker", "traefik"])
|
||||
assert [r["name"] for r in rows] == ["ancient"]
|
||||
assert rows[0]["removable"] is True
|
||||
assert rows[0]["reason"] is None
|
||||
|
||||
|
||||
def test_no_orphans_when_everything_is_installed():
|
||||
assert find_orphaned_plugins(["docker", "snmp"], ["docker", "snmp"]) == []
|
||||
|
||||
|
||||
def test_installed_plugin_without_stored_settings_is_not_an_orphan():
|
||||
# Never configured is not the same as left behind.
|
||||
assert find_orphaned_plugins([], ["docker"]) == []
|
||||
|
||||
|
||||
def test_orphans_are_sorted_and_skip_empty_names():
|
||||
rows = find_orphaned_plugins(["zeta", "", "alpha"], [])
|
||||
assert [r["name"] for r in rows] == ["alpha", "zeta"]
|
||||
|
||||
|
||||
def test_accepts_arbitrary_iterables():
|
||||
# Callers pass a generator of discovered dirs, not a list.
|
||||
rows = find_orphaned_plugins((n for n in ["gone"]), (n for n in ["here"]))
|
||||
assert [r["name"] for r in rows] == ["gone"]
|
||||
|
||||
|
||||
# ── load failures folded in (issue #2638) ────────────────────────────────────
|
||||
|
||||
|
||||
def test_failure_reason_is_attached_to_its_row():
|
||||
rows = find_orphaned_plugins(
|
||||
["ancient"], [], {"ancient": "Plugin directory not found in: ['/app/plugins']"})
|
||||
assert rows[0]["reason"] == "Plugin directory not found in: ['/app/plugins']"
|
||||
|
||||
|
||||
def test_failure_with_no_stored_row_still_gets_a_row():
|
||||
"""The banner counts it, so the page must show it — that was the dead end.
|
||||
|
||||
A plugin enabled by a DEFAULTS entry has no stored row, so a stored-only
|
||||
view would leave the banner pointing at a page with nothing on it.
|
||||
"""
|
||||
rows = find_orphaned_plugins([], [], {"http": "Plugin directory not found"})
|
||||
assert [r["name"] for r in rows] == ["http"]
|
||||
assert rows[0]["reason"] == "Plugin directory not found"
|
||||
|
||||
|
||||
def test_failure_with_no_stored_row_is_not_removable():
|
||||
# Offering Remove would delete nothing while the default reasserts it.
|
||||
rows = find_orphaned_plugins([], [], {"http": "Plugin directory not found"})
|
||||
assert rows[0]["removable"] is False
|
||||
|
||||
|
||||
def test_failure_of_an_installed_plugin_is_not_listed_here():
|
||||
# A discovered plugin that failed already shows the reason on its own card.
|
||||
assert find_orphaned_plugins(
|
||||
["docker"], ["docker"], {"docker": "boom"}) == []
|
||||
|
||||
|
||||
def test_stored_orphan_and_failure_are_not_duplicated():
|
||||
rows = find_orphaned_plugins(["gone"], [], {"gone": "Plugin directory not found"})
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["removable"] is True and rows[0]["reason"]
|
||||
|
||||
|
||||
def test_every_undiscovered_failure_is_represented():
|
||||
# The invariant that keeps the banner count and this list in agreement.
|
||||
failures = {"a": "x", "b": "y", "docker": "z"}
|
||||
rows = find_orphaned_plugins([], ["docker"], failures)
|
||||
listed = {r["name"] for r in rows}
|
||||
assert listed == {"a", "b"}
|
||||
@@ -9,9 +9,7 @@ and a DEFAULTS-merged-with-stored dict, mirroring get_all_settings' merge.
|
||||
from steward.core import settings as settings_module
|
||||
from steward.core.settings import DEFAULTS, to_plugins_cfg
|
||||
|
||||
# NB: no "http" — that plugin was dissolved into the unified Monitor entity, and
|
||||
# its lingering default is what test_plugin_settings_hygiene.py now guards against.
|
||||
DEFAULT_ON = {"docker", "host_agent", "snmp"}
|
||||
DEFAULT_ON = {"docker", "host_agent", "http", "snmp"}
|
||||
VENDOR_OPT_IN = {"traefik", "unifi"}
|
||||
|
||||
|
||||
@@ -34,7 +32,7 @@ def test_stored_choice_overrides_default():
|
||||
cfg = to_plugins_cfg(merged)
|
||||
assert cfg["docker"]["enabled"] is False
|
||||
# untouched defaults remain enabled
|
||||
assert cfg["snmp"]["enabled"] is True
|
||||
assert cfg["http"]["enabled"] is True
|
||||
|
||||
|
||||
def test_defaults_use_plugin_dot_namespace():
|
||||
|
||||
@@ -1,175 +0,0 @@
|
||||
"""Integration: personal API tokens against real Postgres.
|
||||
|
||||
Resolution reads the token and its user's CURRENT role in one query, so a
|
||||
demoted or disabled user's tokens weaken or die with nothing revoked by hand.
|
||||
The routes are driven through the test client so their transactions run for
|
||||
real: a mocked session cannot catch a double-begin (#884).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
async def _make_user(app, role):
|
||||
from steward.models.users import User
|
||||
uid = str(uuid.uuid4())
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=f"tok-{uid[:8]}", email=f"{uid}@test",
|
||||
password_hash="x", role=role, is_active=True))
|
||||
return uid
|
||||
|
||||
|
||||
async def _mint(app, uid, user_role, role, **kw):
|
||||
from steward.core import api_tokens
|
||||
async with app.db_sessionmaker() as s:
|
||||
raw, token = await api_tokens.create_token(
|
||||
s, user_id=uid, user_role=user_role, name="it", role=role, **kw)
|
||||
await s.commit()
|
||||
return raw, token.id
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_resolve_lifecycle(app):
|
||||
from sqlalchemy import select, update
|
||||
from steward.core import api_tokens
|
||||
from steward.models.api_tokens import ApiToken
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
async def _go():
|
||||
uid = await _make_user(app, UserRole.admin)
|
||||
raw, tid = await _mint(app, uid, UserRole.admin, UserRole.operator)
|
||||
|
||||
caller = await api_tokens.resolve_token(app, raw)
|
||||
assert caller is not None
|
||||
assert (caller.user_id, caller.role, caller.token_id) == (uid, UserRole.operator, tid)
|
||||
async with app.db_sessionmaker() as s:
|
||||
used = (await s.execute(select(ApiToken.last_used_at).where(ApiToken.id == tid))).scalar_one()
|
||||
assert used is not None
|
||||
|
||||
# A wrong token, and the right one with a character changed, resolve to nobody.
|
||||
assert await api_tokens.resolve_token(app, api_tokens.generate_token()) is None
|
||||
assert await api_tokens.resolve_token(app, raw[:-1] + ("A" if raw[-1] != "A" else "B")) is None
|
||||
|
||||
# Demoting the user caps the token at the user's new role.
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(update(User).where(User.id == uid).values(role=UserRole.viewer))
|
||||
assert (await api_tokens.resolve_token(app, raw)).role == UserRole.viewer
|
||||
|
||||
# Disabling the user kills the token.
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(update(User).where(User.id == uid).values(is_active=False))
|
||||
assert await api_tokens.resolve_token(app, raw) is None
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(update(User).where(User.id == uid).values(is_active=True))
|
||||
assert await api_tokens.resolve_token(app, raw) is not None
|
||||
|
||||
# Revoking kills it, and only the owner can revoke.
|
||||
other = await _make_user(app, UserRole.admin)
|
||||
async with app.db_sessionmaker() as s:
|
||||
assert await api_tokens.revoke_token(s, user_id=other, token_id=tid) is None
|
||||
await s.commit()
|
||||
assert await api_tokens.resolve_token(app, raw) is not None
|
||||
async with app.db_sessionmaker() as s:
|
||||
assert await api_tokens.revoke_token(s, user_id=uid, token_id=tid) is not None
|
||||
await s.commit()
|
||||
assert await api_tokens.resolve_token(app, raw) is None
|
||||
|
||||
# An expired token resolves to nobody.
|
||||
raw2, tid2 = await _mint(app, uid, UserRole.viewer, UserRole.viewer)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(update(ApiToken).where(ApiToken.id == tid2).values(
|
||||
expires_at=datetime.now(timezone.utc) - timedelta(seconds=1)))
|
||||
assert await api_tokens.resolve_token(app, raw2) is None
|
||||
|
||||
# Deleting the user takes their tokens with them.
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(User.__table__.delete().where(User.id == uid))
|
||||
left = (await s.execute(select(ApiToken).where(ApiToken.user_id == uid))).all()
|
||||
assert left == []
|
||||
|
||||
asyncio.run(_go())
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_routes_create_list_and_revoke(app):
|
||||
from sqlalchemy import select
|
||||
from steward.core import api_tokens
|
||||
from steward.models.api_tokens import ApiToken
|
||||
from steward.models.audit import AuditEvent
|
||||
from steward.models.users import UserRole
|
||||
|
||||
async def _go():
|
||||
uid = await _make_user(app, UserRole.operator)
|
||||
client = app.test_client()
|
||||
async with client.session_transaction() as sess:
|
||||
sess["user_id"] = uid
|
||||
sess["user_role"] = "operator"
|
||||
sess["username"] = "route-test"
|
||||
|
||||
resp = await client.get("/ai-access/")
|
||||
assert resp.status_code == 200
|
||||
page = await resp.get_data(as_text=True)
|
||||
assert "You have no tokens yet" in page
|
||||
# An operator can't offer admin.
|
||||
assert 'value="admin"' not in page
|
||||
# The connect card names the endpoint, with a placeholder for the token.
|
||||
assert 'value="http://localhost/mcp"' in page and "claude mcp add" in page
|
||||
|
||||
resp = await client.post("/ai-access/tokens", form={
|
||||
"name": "laptop", "role": "admin", "expires_in_days": ""})
|
||||
assert resp.status_code == 400
|
||||
assert "higher role" in await resp.get_data(as_text=True)
|
||||
|
||||
resp = await client.post("/ai-access/tokens", form={
|
||||
"name": "laptop", "role": "operator", "expires_in_days": "30"})
|
||||
assert resp.status_code == 200
|
||||
created = await resp.get_data(as_text=True)
|
||||
raw = re.search(r'value="(stw_[^"]+)"', created).group(1)
|
||||
# The recipes on the created page carry the new token, ready to paste.
|
||||
assert f"Bearer {raw}" in created and "mcp-remote" in created
|
||||
caller = await api_tokens.resolve_token(app, raw)
|
||||
assert caller is not None and caller.role == UserRole.operator
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
token = (await s.execute(select(ApiToken).where(ApiToken.user_id == uid))).scalar_one()
|
||||
page = await (await client.get("/ai-access/")).get_data(as_text=True)
|
||||
assert "laptop" in page and token.prefix in page
|
||||
assert raw not in page
|
||||
|
||||
resp = await client.post(f"/ai-access/tokens/{token.id}/revoke")
|
||||
assert resp.status_code == 302
|
||||
assert await api_tokens.resolve_token(app, raw) is None
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
actions = (await s.execute(
|
||||
select(AuditEvent.action).where(AuditEvent.user_id == uid))).scalars().all()
|
||||
assert "api_token.create" in actions and "api_token.revoke" in actions
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -7,7 +7,6 @@ containers on different hosts no longer collide. Requires STEWARD_DATABASE_URL.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
@@ -76,28 +75,6 @@ def test_events_and_swarm_tables_exist(app):
|
||||
asyncio.run(_go())
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_docker_logs_table_shape(app):
|
||||
"""docker_009 created docker_logs: host-scoped, with the twin indexes the
|
||||
viewer (host, container, ts) and the age-cutoff prune (ts) rely on."""
|
||||
from sqlalchemy import text
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
cols = {r[0] for r in (await s.execute(text(
|
||||
"SELECT column_name FROM information_schema.columns "
|
||||
"WHERE table_name = 'docker_logs'"))).all()}
|
||||
assert cols, "docker_logs missing entirely"
|
||||
assert {"id", "host_id", "container_name", "ts", "stream", "line"} <= cols
|
||||
idx = {r[0] for r in (await s.execute(text(
|
||||
"SELECT indexname FROM pg_indexes "
|
||||
"WHERE tablename = 'docker_logs'"))).all()}
|
||||
assert "ix_docker_logs_host_container_time" in idx
|
||||
assert "ix_docker_logs_ts" in idx
|
||||
|
||||
asyncio.run(_go())
|
||||
|
||||
|
||||
def _persist_fn(app):
|
||||
"""Resolve persist_host_docker via the registered capability if the docker
|
||||
plugin is loaded, else import it directly (the import is safe only when the
|
||||
@@ -173,97 +150,6 @@ def test_persist_scopes_containers_by_host(app):
|
||||
assert enrich == ("healthy", 2, "web", 1000) # enrichment round-trips
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_persist_logs_stores_lines(app):
|
||||
"""Pushed container logs land in docker_logs, host-scoped, one row per line,
|
||||
ordered by their own ts; the agent's _steward truncation marker is dropped."""
|
||||
from sqlalchemy import text
|
||||
from steward.models.hosts import Host
|
||||
|
||||
persist = _persist_fn(app)
|
||||
now = datetime.now(timezone.utc)
|
||||
log_batches = [(now, [
|
||||
{"container": "web", "stream": "stdout",
|
||||
"ts": "2026-01-01T00:00:01+00:00", "line": "started"},
|
||||
{"container": "web", "stream": "stderr",
|
||||
"ts": "2026-01-01T00:00:02+00:00", "line": "warn: x"},
|
||||
{"container": "_steward", "stream": "stderr", "ts": None, "line": "truncated"},
|
||||
])]
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(text("DELETE FROM docker_logs"))
|
||||
h = Host(id=str(uuid.uuid4()), name="loghost", address="10.0.0.9")
|
||||
s.add(h)
|
||||
await s.flush()
|
||||
hid = h.id
|
||||
# snapshots empty; logs passed as the 6th positional arg.
|
||||
await persist(s, h, [], None, None, log_batches)
|
||||
rows = (await s.execute(text(
|
||||
"SELECT container_name, stream, line FROM docker_logs "
|
||||
"WHERE host_id = :h ORDER BY ts"), {"h": hid})).all()
|
||||
return [tuple(r) for r in rows]
|
||||
|
||||
assert asyncio.run(_go()) == [
|
||||
("web", "stdout", "started"),
|
||||
("web", "stderr", "warn: x"),
|
||||
]
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_persist_logs_respects_toggle_and_exclude(app):
|
||||
"""Server-side controls (Settings): an excluded container's lines are dropped
|
||||
while others persist; the global toggle off stores nothing new."""
|
||||
from types import SimpleNamespace
|
||||
from sqlalchemy import text
|
||||
from steward.models.hosts import Host
|
||||
from steward.core.settings import set_setting
|
||||
|
||||
persist = _persist_fn(app)
|
||||
now = datetime.now(timezone.utc)
|
||||
batch = [(now, [
|
||||
{"container": "web", "stream": "stdout", "ts": None, "line": "keep-me"},
|
||||
{"container": "noisy", "stream": "stdout", "ts": None, "line": "drop-me"},
|
||||
])]
|
||||
|
||||
hid = str(uuid.uuid4())
|
||||
# Stand-in host: the logs-only persist path reads only host.id, so a plain
|
||||
# object sidesteps ORM attribute-expiry across the commit boundaries below.
|
||||
host = SimpleNamespace(id=hid, name="loghost3")
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
# Back-to-back begin blocks with NO read in between — a SELECT between
|
||||
# them would autobegin a txn and make the next begin() collide.
|
||||
async with s.begin():
|
||||
await s.execute(text("DELETE FROM docker_logs"))
|
||||
await set_setting(s, "docker.logs.enabled", True)
|
||||
await set_setting(s, "docker.logs.exclude", ["noisy"])
|
||||
s.add(Host(id=hid, name="loghost3", address="10.7.7.10"))
|
||||
await s.flush() # host row before its log rows (FK order)
|
||||
await persist(s, host, [], None, None, batch) # web kept, noisy excluded
|
||||
async with s.begin(): # global kill-switch off → nothing stored
|
||||
await set_setting(s, "docker.logs.enabled", False)
|
||||
await persist(s, host, [], None, None, batch)
|
||||
async with s.begin(): # restore defaults for other tests
|
||||
await set_setting(s, "docker.logs.enabled", True)
|
||||
await set_setting(s, "docker.logs.exclude", [])
|
||||
# Reads only after the last begin block (they autobegin, but nothing
|
||||
# opens a begin() after them). Toggle-off added nothing, so the state
|
||||
# here still reflects the exclude phase.
|
||||
kept = {r[0] for r in (await s.execute(text(
|
||||
"SELECT DISTINCT container_name FROM docker_logs WHERE host_id=:h"),
|
||||
{"h": hid})).all()}
|
||||
after_off = (await s.execute(text(
|
||||
"SELECT COUNT(*) FROM docker_logs WHERE host_id=:h"), {"h": hid})).scalar()
|
||||
return kept, after_off
|
||||
|
||||
kept, after_off = asyncio.run(_go())
|
||||
assert kept == {"web"} # excluded 'noisy' dropped on ingest
|
||||
assert after_off == 1 # toggle off added nothing → still just the one 'web' line
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_large_memory_values_persist_as_bigint(app):
|
||||
"""A container using >2^31 bytes of RAM must persist. Regression: mem_usage_bytes
|
||||
@@ -392,11 +278,7 @@ def test_swarm_topology_persisted(app):
|
||||
swarm = {
|
||||
"services": [{"service_name": "web", "mode": "replicated",
|
||||
"desired": 3, "running": 2, "image": "nginx",
|
||||
"placement": [{"node_id": "n1", "running": 2}],
|
||||
"tasks": [{"task_id": "t1", "slot": 3, "node_id": "n1",
|
||||
"desired_state": "shutdown", "state": "failed",
|
||||
"message": "started", "error": "task: non-zero exit (1)",
|
||||
"exit_code": 1, "at": "2026-10-06T13:19:51Z"}]}],
|
||||
"placement": [{"node_id": "n1", "running": 2}]}],
|
||||
"nodes": [{"node_id": "n1", "hostname": "mgr", "role": "manager",
|
||||
"availability": "active", "status": "ready", "leader": True}],
|
||||
}
|
||||
@@ -413,7 +295,7 @@ def test_swarm_topology_persisted(app):
|
||||
await persist(s, h, [], swarm)
|
||||
hid = h.id
|
||||
svc = (await s.execute(text(
|
||||
"SELECT mode, desired, running, image, placement_json, tasks_json "
|
||||
"SELECT mode, desired, running, image, placement_json "
|
||||
"FROM docker_swarm_services WHERE host_id = :h"), {"h": hid})).first()
|
||||
node = (await s.execute(text(
|
||||
"SELECT role, availability, status, leader "
|
||||
@@ -423,8 +305,6 @@ def test_swarm_topology_persisted(app):
|
||||
svc, node = asyncio.run(_go())
|
||||
assert svc[0] == "replicated" and svc[1] == 3 and svc[2] == 2 and svc[3] == "nginx"
|
||||
assert "n1" in svc[4] # placement JSON carries the node id
|
||||
(task,) = json.loads(svc[5]) # docker_010: the manager's task records survive
|
||||
assert task["error"] == "task: non-zero exit (1)" and task["slot"] == 3
|
||||
assert node[0] == "manager" and node[2] == "ready" and node[3] is True
|
||||
|
||||
|
||||
@@ -571,67 +451,6 @@ def test_retention_rollup_and_prune(app):
|
||||
assert counts["events_pruned"] == 1 and counts["rollup_pruned"] == 1
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_retention_logs_ring(app):
|
||||
"""docker_logs ring (m79): lines past the age window are pruned; within it,
|
||||
each container keeps only the newest ~cap bytes; containers are independent."""
|
||||
from datetime import timedelta
|
||||
from sqlalchemy import text
|
||||
from steward.models.hosts import Host
|
||||
|
||||
run_retention = _retention_fn(app)
|
||||
now = datetime(2026, 6, 19, 12, 0, 0, tzinfo=timezone.utc)
|
||||
line40 = "x" * 40 # 40 chars/line; cap=100 keeps 3 lines (excl-prefix 0/40/80)
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(text("DELETE FROM docker_logs"))
|
||||
h = Host(id=str(uuid.uuid4()), name="loghost2", address="10.7.7.8")
|
||||
s.add(h)
|
||||
await s.flush()
|
||||
hid = h.id
|
||||
|
||||
def ins(cn, ts, line):
|
||||
return s.execute(text(
|
||||
"INSERT INTO docker_logs "
|
||||
"(id, host_id, container_name, ts, stream, line) "
|
||||
"VALUES (:id,:h,:cn,:ts,'stdout',:line)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "cn": cn,
|
||||
"ts": ts, "line": line})
|
||||
|
||||
# Past the 3-day age window → age-pruned.
|
||||
await ins("old", now - timedelta(days=10), "z")
|
||||
# 5 recent 'web' lines (40 bytes each = 200 > cap 100) → keep 3.
|
||||
for i in range(1, 6):
|
||||
await ins("web", now - timedelta(minutes=6 - i), line40)
|
||||
# 2 recent 'db' lines (80 bytes ≤ cap) → both survive (isolation).
|
||||
await ins("db", now - timedelta(minutes=2), line40)
|
||||
await ins("db", now - timedelta(minutes=1), line40)
|
||||
|
||||
async with s.begin():
|
||||
counts = await run_retention(
|
||||
s, events_days=30, metrics_raw_days=7, metrics_rollup_days=90,
|
||||
logs_retention_days=3, logs_max_bytes_per_container=100, now=now,
|
||||
)
|
||||
|
||||
def n(cn):
|
||||
return s.execute(text(
|
||||
"SELECT COUNT(*) FROM docker_logs WHERE host_id=:h AND container_name=:cn"),
|
||||
{"h": hid, "cn": cn})
|
||||
web = (await n("web")).scalar()
|
||||
db = (await n("db")).scalar()
|
||||
old = (await n("old")).scalar()
|
||||
return counts, web, db, old
|
||||
|
||||
counts, web, db, old = asyncio.run(_go())
|
||||
assert old == 0 # age window
|
||||
assert web == 3 # newest ~100 bytes kept, oldest 2 dropped
|
||||
assert db == 2 # under cap → untouched (per-container)
|
||||
assert counts["logs_age_pruned"] == 1
|
||||
assert counts["logs_size_pruned"] == 2
|
||||
|
||||
|
||||
def test_widget_dedup_collapses_cross_manager_duplicates():
|
||||
"""The same swarm task is reported by every manager (identical container_id);
|
||||
the dashboard widget must count it once. Older agents send no container_id,
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
"""Integration: the /mcp endpoint end to end against real Postgres.
|
||||
|
||||
A token is minted through the real service, then MCP requests go through the
|
||||
app's ASGI callable (the same path a client's HTTP request takes): the
|
||||
database gate, bearer resolution and role filtering all run for real.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.mcp._http import HEADERS as _HEADERS, asgi_request, rpc, rpc_result
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_mcp_over_http_with_a_minted_token(app):
|
||||
from steward.core import api_tokens
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
async def _go():
|
||||
uid = str(uuid.uuid4())
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=f"mcp-{uid[:8]}", email=f"{uid}@test",
|
||||
password_hash="x", role=UserRole.admin, is_active=True))
|
||||
async with app.db_sessionmaker() as s:
|
||||
raw, token = await api_tokens.create_token(
|
||||
s, user_id=uid, user_role=UserRole.admin, name="it", role=UserRole.viewer)
|
||||
await s.commit()
|
||||
|
||||
auth = {**_HEADERS, "Authorization": f"Bearer {raw}"}
|
||||
async with app.mcp_server.session_manager.run():
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc("initialize", {
|
||||
"protocolVersion": "2025-06-18", "capabilities": {},
|
||||
"clientInfo": {"name": "it", "version": "0"}}))
|
||||
assert status == 200, body
|
||||
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc("tools/list", id_=2))
|
||||
assert status == 200, body
|
||||
assert "whoami" in {t["name"] for t in rpc_result(body)["result"]["tools"]}
|
||||
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc(
|
||||
"tools/call", {"name": "whoami", "arguments": {}}, id_=3))
|
||||
assert status == 200, body
|
||||
me = json.loads(rpc_result(body)["result"]["content"][0]["text"])
|
||||
# The token's viewer ceiling wins over the user's admin role.
|
||||
assert (me["user_id"], me["role"], me["token_id"]) == (uid, "viewer", token.id)
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
await api_tokens.revoke_token(s, user_id=uid, token_id=token.id)
|
||||
await s.commit()
|
||||
status, _, _ = await asgi_request(app, "POST", "/mcp", auth, rpc("tools/list", id_=4))
|
||||
assert status == 401
|
||||
|
||||
asyncio.run(_go())
|
||||
|
||||
|
||||
def _tool_result(body) -> dict:
|
||||
return rpc_result(body)["result"]
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_roles_reads_and_audited_writes_over_http(app):
|
||||
"""Over HTTP: a viewer sees no write tools and is refused one by name; an
|
||||
operator reads get_overview and makes an audited change."""
|
||||
from sqlalchemy import select
|
||||
from steward.core import api_tokens
|
||||
from steward.models.audit import AuditEvent
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
async def _go():
|
||||
uid = str(uuid.uuid4())
|
||||
username = f"mcp-op-{uid[:8]}"
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=username, email=f"{uid}@test",
|
||||
password_hash="x", role=UserRole.operator, is_active=True))
|
||||
async with app.db_sessionmaker() as s:
|
||||
viewer_raw, _ = await api_tokens.create_token(
|
||||
s, user_id=uid, user_role=UserRole.operator, name="ro", role=UserRole.viewer)
|
||||
op_raw, op_token = await api_tokens.create_token(
|
||||
s, user_id=uid, user_role=UserRole.operator, name="rw", role=UserRole.operator)
|
||||
await s.commit()
|
||||
viewer = {**_HEADERS, "Authorization": f"Bearer {viewer_raw}"}
|
||||
operator = {**_HEADERS, "Authorization": f"Bearer {op_raw}"}
|
||||
|
||||
async with app.mcp_server.session_manager.run():
|
||||
_, _, body = await asgi_request(app, "POST", "/mcp", viewer, rpc("tools/list", id_=1))
|
||||
names = {t["name"] for t in _tool_result(body)["tools"]}
|
||||
assert "get_overview" in names and "list_playbooks" in names
|
||||
assert not names & {"run_playbook", "create_monitor", "delete_alert_rule"}
|
||||
|
||||
_, _, body = await asgi_request(app, "POST", "/mcp", viewer, rpc(
|
||||
"tools/call", {"name": "create_maintenance_window",
|
||||
"arguments": {"name": "x", "duration_minutes": 5}}, id_=2))
|
||||
refused = _tool_result(body) if "result" in rpc_result(body) else rpc_result(body)["error"]
|
||||
assert "needs the operator role" in json.dumps(refused)
|
||||
|
||||
_, _, body = await asgi_request(app, "POST", "/mcp", operator, rpc(
|
||||
"tools/call", {"name": "get_overview", "arguments": {}}, id_=3))
|
||||
result = _tool_result(body)
|
||||
assert not result.get("isError"), result
|
||||
overview = json.loads(result["content"][0]["text"])
|
||||
assert {"generated_at", "all_clear", "counts", "findings"} <= set(overview)
|
||||
|
||||
name = f"it-window-{uid[:8]}"
|
||||
_, _, body = await asgi_request(app, "POST", "/mcp", operator, rpc(
|
||||
"tools/call", {"name": "create_maintenance_window",
|
||||
"arguments": {"name": name, "duration_minutes": 5}}, id_=4))
|
||||
result = _tool_result(body)
|
||||
assert not result.get("isError"), result
|
||||
window_id = json.loads(result["content"][0]["text"])["window_id"]
|
||||
_, _, body = await asgi_request(app, "POST", "/mcp", operator, rpc(
|
||||
"tools/call", {"name": "end_maintenance_window",
|
||||
"arguments": {"window": window_id}}, id_=5))
|
||||
assert not _tool_result(body).get("isError")
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
rows = list((await s.execute(select(AuditEvent).where(
|
||||
AuditEvent.user_id == uid, AuditEvent.entity_id == name))).scalars())
|
||||
assert sorted(e.action for e in rows) == ["maintenance_window.created",
|
||||
"maintenance_window.ended"]
|
||||
assert {e.username for e in rows} == {f"{username} via MCP"}
|
||||
assert all(json.loads(e.detail_json)["token_id"] == op_token.id for e in rows)
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -1,199 +0,0 @@
|
||||
"""Integration: the automation tools against real Postgres and a real ansible-playbook.
|
||||
|
||||
One local playbook (connection: local) writes a file. A run started through
|
||||
run_playbook is check mode unless told otherwise, so the first run must leave
|
||||
no file behind and the confirmed apply must write it. Every write is audited as
|
||||
"<user> via MCP", and the secret variable reaches the play without being
|
||||
stored, audited or echoed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import textwrap
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
SECRET = "s3cr3t-value-xyz"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_runs_schedules_and_alert_actions_through_mcp(app, tmp_path):
|
||||
from sqlalchemy import select
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.mcp import _context
|
||||
from steward.mcp.tools import automation as a
|
||||
from steward.models.alerts import AlertOperator, AlertRule
|
||||
from steward.models.ansible import AnsibleRun
|
||||
from steward.models.ansible_inventory import AnsibleTarget
|
||||
from steward.models.audit import AuditEvent
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
src, uid, username = f"mcp-src-{tag}", str(uuid.uuid4()), f"mcp-op-{tag}"
|
||||
target_name, rule_name = f"mcp-target-{tag}", f"mcp-rule-{tag}"
|
||||
out_file = tmp_path / "applied.txt"
|
||||
|
||||
play_dir = tmp_path / "playbooks"
|
||||
play_dir.mkdir()
|
||||
(play_dir / "play.yml").write_text(textwrap.dedent("""\
|
||||
# description: Writes a marker file (MCP integration test)
|
||||
# steward:confirm: true
|
||||
- hosts: all
|
||||
connection: local
|
||||
gather_facts: false
|
||||
vars:
|
||||
greeting: hello
|
||||
api_token: changeme
|
||||
out_file: /nonexistent
|
||||
tasks:
|
||||
- debug:
|
||||
msg: "GREETING-{{ greeting }} TOKEN-LEN-{{ api_token | length }}"
|
||||
- copy:
|
||||
dest: "{{ out_file }}"
|
||||
content: applied
|
||||
"""))
|
||||
(play_dir / "hosts").write_text("localhost ansible_connection=local\n")
|
||||
app.config["ANSIBLE"] = {"sources": [{"name": src, "type": "local", "path": str(play_dir)}]}
|
||||
|
||||
async def _seed():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=username, email=f"{tag}@e.test",
|
||||
password_hash="x", role=UserRole.admin, is_active=True))
|
||||
async with s.begin():
|
||||
s.add(AnsibleTarget(name=target_name, address="127.0.0.1",
|
||||
ansible_vars={"ansible_connection": "local"}))
|
||||
s.add(AlertRule(name=rule_name, source_module="host_agent", resource_name="nas",
|
||||
metric_name="cpu_pct", operator=AlertOperator.gt, threshold=90.0,
|
||||
created_by=uid))
|
||||
|
||||
async def _audit(action, entity_id=None):
|
||||
async with app.db_sessionmaker() as s:
|
||||
stmt = select(AuditEvent).where(AuditEvent.action == action,
|
||||
AuditEvent.user_id == uid)
|
||||
if entity_id:
|
||||
stmt = stmt.where(AuditEvent.entity_id == entity_id)
|
||||
return list((await s.execute(stmt)).scalars())
|
||||
|
||||
async def _go():
|
||||
await _seed()
|
||||
_context._app_ctx.set(app)
|
||||
_context._caller_ctx.set(Caller(user_id=uid, username=username,
|
||||
role=UserRole.admin, token_id=42))
|
||||
|
||||
# ── discovery ──────────────────────────────────────────────────────
|
||||
books = (await a.list_playbooks(source=src))["playbooks"]
|
||||
assert [(b["playbook"], b["confirm"]) for b in books] == [("play.yml", True)]
|
||||
pb = await a.get_playbook(src, "play.yml")
|
||||
defaults = {v["name"]: (v["default"], v["secret"]) for v in pb["variables"]}
|
||||
assert defaults["api_token"] == ("<secret>", True)
|
||||
assert defaults["greeting"] == ("hello", False)
|
||||
inv = await a.list_inventory()
|
||||
assert any(t["name"] == target_name and t["var_names"] == ["ansible_connection"]
|
||||
for t in inv["targets"])
|
||||
with pytest.raises(ValueError, match="has no playbook 'nope.yml'"):
|
||||
await a.get_playbook(src, "nope.yml")
|
||||
|
||||
# ── a check-mode run is the default and changes nothing ───────────
|
||||
extra = {"greeting": "yo", "api_token": SECRET, "out_file": str(out_file)}
|
||||
started = await a.run_playbook(src, "play.yml", scope=f"target:{target_name}",
|
||||
extra_vars=extra)
|
||||
assert started["check"] is True and started["secret_vars_passed"] == ["api_token"]
|
||||
assert SECRET not in json.dumps(started)
|
||||
done = await a.wait_for_run(started["run_id"], timeout_s=120)
|
||||
assert done["finished"] and done["status"] == "success", done["tail"]
|
||||
assert done["check"] is True and done["triggered_by"] == username
|
||||
# The scope reads as the target's name, not its id.
|
||||
assert done["scope"].startswith("steward:target:") and done["scope_label"] == f"target {target_name}"
|
||||
assert not out_file.exists()
|
||||
log = "\n".join(done["tail"])
|
||||
assert "GREETING-yo" in log and f"TOKEN-LEN-{len(SECRET)}" in log
|
||||
assert SECRET not in json.dumps(done)
|
||||
async with app.db_sessionmaker() as s:
|
||||
run = await s.get(AnsibleRun, started["run_id"])
|
||||
assert run.params["check"] is True and SECRET not in json.dumps(run.params)
|
||||
|
||||
rows = await _audit("ansible.run.started", started["run_id"])
|
||||
assert len(rows) == 1 and rows[0].username == f"{username} via MCP"
|
||||
detail = json.loads(rows[0].detail_json)
|
||||
assert detail["check"] is True and detail["token_id"] == 42
|
||||
assert detail["secret_vars"] == ["api_token"] and SECRET not in rows[0].detail_json
|
||||
assert detail["scope_label"].startswith(f"target {target_name}")
|
||||
|
||||
# ── applying a confirm playbook needs confirmed=true ──────────────
|
||||
with pytest.raises(ValueError, match="flagged destructive"):
|
||||
await a.run_playbook(src, "play.yml", scope=f"target:{target_name}",
|
||||
extra_vars=extra, check=False)
|
||||
applied = await a.run_playbook(src, "play.yml", scope=f"target:{target_name}",
|
||||
extra_vars=extra, check=False, confirmed=True)
|
||||
done = await a.wait_for_run(applied["run_id"], timeout_s=120)
|
||||
assert done["status"] == "success" and done["check"] is False, done["tail"]
|
||||
assert out_file.read_text() == "applied"
|
||||
|
||||
listed = (await a.list_ansible_runs(playbook="play.yml", range="1h", limit=500))["runs"]
|
||||
ours = {r["run_id"]: r for r in listed}
|
||||
assert ours[started["run_id"]]["check"] is True
|
||||
assert ours[applied["run_id"]]["check"] is False
|
||||
grep = await a.get_ansible_run(applied["run_id"], grep="greeting-")
|
||||
assert grep["matches_total"] >= 1
|
||||
with pytest.raises(ValueError, match="already ended"):
|
||||
await a.cancel_ansible_run(applied["run_id"])
|
||||
|
||||
with pytest.raises(ValueError, match="scope 'everything'"):
|
||||
await a.run_playbook(src, "play.yml", scope="everything")
|
||||
with pytest.raises(ValueError, match="no target named"):
|
||||
await a.run_playbook(src, "play.yml", scope=f"target:ghost-{tag}")
|
||||
|
||||
# ── schedules ──────────────────────────────────────────────────────
|
||||
with pytest.raises(ValueError, match="api_token is secret.*Nothing was saved"):
|
||||
await a.create_schedule(f"sched-{tag}", src, "play.yml", 3600,
|
||||
scope=f"target:{target_name}", extra_vars=extra)
|
||||
sc = await a.create_schedule(f"sched-{tag}", src, "play.yml", 3600,
|
||||
scope=f"target:{target_name}",
|
||||
extra_vars={"out_file": str(out_file)})
|
||||
assert sc["check"] is True and sc["enabled"] is True and sc["scope_label"] == f"target {target_name}"
|
||||
up = await a.update_schedule(f"sched-{tag}", enabled=False, interval_s=86400)
|
||||
assert (up["enabled"], up["interval_s"], up["check"]) == (False, 86400, True)
|
||||
assert up["extra_vars"] == {"out_file": str(out_file)} # untouched fields stay
|
||||
assert any(s["schedule_id"] == sc["schedule_id"] for s in (await a.list_schedules())["schedules"])
|
||||
await a.delete_schedule(sc["schedule_id"])
|
||||
assert not any(s["schedule_id"] == sc["schedule_id"]
|
||||
for s in (await a.list_schedules())["schedules"])
|
||||
for action in ("ansible.schedule.created", "ansible.schedule.updated",
|
||||
"ansible.schedule.deleted"):
|
||||
assert len(await _audit(action, sc["schedule_id"])) == 1, action
|
||||
|
||||
# ── alert actions ──────────────────────────────────────────────────
|
||||
with pytest.raises(ValueError, match="no inventory file 'missing'"):
|
||||
await a.set_alert_action(rule_name, src, "play.yml", "missing")
|
||||
act = await a.set_alert_action(rule_name, src, "play.yml", "hosts",
|
||||
extra_vars={"greeting": "alert"})
|
||||
assert act["action"] == {"source": src, "playbook": "play.yml", "inventory": "hosts",
|
||||
"extra_vars": ["greeting=alert"], "check": True}
|
||||
cleared = await a.set_alert_action(rule_name)
|
||||
assert cleared["action"] is None
|
||||
async with app.db_sessionmaker() as s:
|
||||
rule = (await s.execute(select(AlertRule).where(AlertRule.name == rule_name))).scalar_one()
|
||||
assert rule.ansible_action is None
|
||||
assert len(await _audit("alert_rule.action_set", rule_name)) == 1
|
||||
assert len(await _audit("alert_rule.action_cleared", rule_name)) == 1
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -1,146 +0,0 @@
|
||||
"""Integration: the monitoring-config write tools against real Postgres.
|
||||
|
||||
A monitor is created on a host, edited (config merged, a rename warning for
|
||||
the rule that watches it by name), paused and deleted; a rule is created on
|
||||
that monitor, edited, acknowledged once firing, and deleted; a maintenance
|
||||
window in progress is ended and an upcoming one cancelled. Every write leaves
|
||||
an audit row as "<user> via MCP".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_monitor_rule_ack_and_maintenance_writes(app):
|
||||
from sqlalchemy import select
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.mcp import _context
|
||||
from steward.mcp.tools import alerts as a
|
||||
from steward.mcp.tools import monitors as m
|
||||
from steward.models.alerts import AlertRule, AlertState, AlertStateEnum, MaintenanceWindow
|
||||
from steward.models.audit import AuditEvent
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.monitors import Monitor
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
uid, username, host_name = str(uuid.uuid4()), f"mcp-cfg-{tag}", f"cfg-host-{tag}"
|
||||
mon_name, rule_name = f"web-{tag}", f"web-down-{tag}"
|
||||
|
||||
async def _seed():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=username, email=f"{tag}@e.test",
|
||||
password_hash="x", role=UserRole.admin, is_active=True))
|
||||
s.add(Host(id=str(uuid.uuid4()), name=host_name, address="10.7.7.7"))
|
||||
|
||||
async def _actions():
|
||||
async with app.db_sessionmaker() as s:
|
||||
rows = (await s.execute(select(AuditEvent).where(AuditEvent.user_id == uid))).scalars()
|
||||
return [(e.action, e.username) for e in rows]
|
||||
|
||||
async def _go():
|
||||
await _seed()
|
||||
_context._app_ctx.set(app)
|
||||
_context._caller_ctx.set(Caller(user_id=uid, username=username,
|
||||
role=UserRole.admin, token_id=7))
|
||||
|
||||
# ── monitors ───────────────────────────────────────────────────────
|
||||
with pytest.raises(ValueError, match="does not take port"):
|
||||
await m.create_monitor(mon_name, "http", "nas.lan", config={"port": 80})
|
||||
with pytest.raises(ValueError, match="no host named"):
|
||||
await m.create_monitor(mon_name, "http", "nas.lan", host=f"ghost-{tag}")
|
||||
made = await m.create_monitor(mon_name, "http", "nas.lan", host=host_name,
|
||||
config={"expected_status": 204,
|
||||
"headers": {"Authorization": "Bearer s3cret"}})
|
||||
assert made["target"] == "https://nas.lan" and made["host_name"] == host_name
|
||||
assert made["config"]["headers"] == {"Authorization": "<redacted>"}
|
||||
assert made["config"]["verify_ssl"] is True
|
||||
|
||||
# ── rules ──────────────────────────────────────────────────────────
|
||||
opts = await a.get_alert_rule_options("http")
|
||||
assert mon_name in opts["resources"] and "is_up" in opts["metrics"]
|
||||
with pytest.raises(ValueError, match="does not report 'up'"):
|
||||
await a.create_alert_rule(rule_name, "http", mon_name, "up", "<", 1)
|
||||
rule = await a.create_alert_rule(rule_name, "http", mon_name, "is_up", "<", 1,
|
||||
consecutive_checks=3)
|
||||
assert rule["condition"] == "is_up < 1 for 3 consecutive checks"
|
||||
rule = await a.update_alert_rule(rule_name, threshold=0.5, enabled=False)
|
||||
assert rule["condition"].startswith("is_up < 0.5") and rule["enabled"] is False
|
||||
|
||||
# Editing the config merges: the stored header survives a timeout change.
|
||||
upd = await m.update_monitor(mon_name, config={"timeout_seconds": 5})
|
||||
assert upd["config"]["timeout_seconds"] == 5 and upd["config"]["expected_status"] == 204
|
||||
async with app.db_sessionmaker() as s:
|
||||
row = await s.get(Monitor, made["monitor_id"])
|
||||
assert row.config["headers"] == {"Authorization": "Bearer s3cret"}
|
||||
|
||||
# A rename leaves the rule on the old name, and says so.
|
||||
renamed = await m.update_monitor(mon_name, name=f"{mon_name}-2")
|
||||
assert rule_name in renamed["warning"]
|
||||
await a.update_alert_rule(rule_name, resource=f"{mon_name}-2")
|
||||
assert "warning" not in await m.update_monitor(f"{mon_name}-2", check_interval_s=30)
|
||||
assert (await m.set_monitor_enabled(f"{mon_name}-2", False))["enabled"] is False
|
||||
|
||||
# ── acknowledge ────────────────────────────────────────────────────
|
||||
with pytest.raises(ValueError, match="not firing"):
|
||||
await a.acknowledge_alert(rule_name)
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
st = (await s.execute(select(AlertState).where(
|
||||
AlertState.rule_id == rule["rule_id"]))).scalar_one()
|
||||
st.state = AlertStateEnum.firing
|
||||
ack = await a.acknowledge_alert(rule_name)
|
||||
assert ack["state"] == "acknowledged"
|
||||
|
||||
# ── maintenance ────────────────────────────────────────────────────
|
||||
win = await a.create_maintenance_window(f"patch-{tag}", duration_minutes=30,
|
||||
source_module="http", resource=f"{mon_name}-2")
|
||||
assert win["active"] is True
|
||||
ended = await a.end_maintenance_window(f"patch-{tag}")
|
||||
assert ended["outcome"] == "ended" and ended["active"] is True
|
||||
with pytest.raises(ValueError, match="already ended"):
|
||||
await a.end_maintenance_window(win["window_id"])
|
||||
later = await a.create_maintenance_window(f"later-{tag}", start_at="2099-01-01T00:00:00Z",
|
||||
end_at="2099-01-01T02:00:00Z")
|
||||
assert (await a.end_maintenance_window(later["window_id"]))["outcome"] == "cancelled"
|
||||
async with app.db_sessionmaker() as s:
|
||||
assert await s.get(MaintenanceWindow, later["window_id"]) is None
|
||||
assert await s.get(MaintenanceWindow, win["window_id"]) is not None
|
||||
|
||||
# ── deletes ────────────────────────────────────────────────────────
|
||||
await a.delete_alert_rule(rule_name)
|
||||
await m.delete_monitor(f"{mon_name}-2")
|
||||
async with app.db_sessionmaker() as s:
|
||||
assert await s.get(AlertRule, rule["rule_id"]) is None
|
||||
assert await s.get(Monitor, made["monitor_id"]) is None
|
||||
|
||||
actions = await _actions()
|
||||
assert all(user == f"{username} via MCP" for _, user in actions)
|
||||
assert sorted({act for act, _ in actions}) == sorted({
|
||||
"monitor.created", "monitor.updated", "monitor.disabled", "monitor.deleted",
|
||||
"alert_rule.created", "alert_rule.updated", "alert_rule.acknowledged",
|
||||
"alert_rule.deleted", "maintenance_window.created", "maintenance_window.ended",
|
||||
"maintenance_window.cancelled"})
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -1,280 +0,0 @@
|
||||
"""Integration: the docker and host_agent plugins' MCP tools against real Postgres.
|
||||
|
||||
Rows are seeded with plain SQL so this file never imports a plugin's models
|
||||
(a second import of them after the app loaded the plugin raises "Table already
|
||||
defined"). The tool modules come from the loaded plugin when there is one,
|
||||
and from the source tree when the plugin is not loaded.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
def _plugin(name):
|
||||
from steward.core.plugin_manager import get_loaded_plugin_modules
|
||||
loaded = get_loaded_plugin_modules().get(name)
|
||||
return loaded if loaded is not None else importlib.import_module(f"plugins.{name}")
|
||||
|
||||
|
||||
def _sub(plugin_module, sub):
|
||||
return importlib.import_module(f"{plugin_module.__name__}.{sub}")
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_docker_and_agent_tools_over_seeded_rows(app):
|
||||
from sqlalchemy import text
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.mcp import _context
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.users import UserRole
|
||||
|
||||
docker, agent = _plugin("docker"), _plugin("host_agent")
|
||||
dtools, atools = _sub(docker, "mcp_tools"), _sub(agent, "mcp_tools")
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
hid, hid2 = str(uuid.uuid4()), str(uuid.uuid4())
|
||||
host_name, quiet_host = f"dock-{tag}", f"quiet-{tag}"
|
||||
web, db_, job, svc = f"web-{tag}", f"db-{tag}", f"job-{tag}", f"svc-{tag}"
|
||||
replaced = f"{svc}.1.old"
|
||||
removed = f"{svc}.2.gone{tag}" # a dead task Docker removed: logs and events only
|
||||
|
||||
def _ago(minutes):
|
||||
return (now - timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
|
||||
|
||||
container_sql = text("""
|
||||
INSERT INTO docker_containers
|
||||
(host_id, name, container_id, image, status, cpu_pct, mem_pct, mem_usage_bytes,
|
||||
mem_limit_bytes, restart_count, ports_json, started_at, scraped_at, health,
|
||||
exit_code, oom_killed, compose_project, service_name, task_id)
|
||||
VALUES (:h, :n, :cid, 'img:1', :st, 5.0, 10.0, 1000, 10000, :rc, '[]', :started, :now,
|
||||
:health, :exit, false, :proj, :svc, :task)
|
||||
""")
|
||||
|
||||
async def _seed():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(Host(id=hid, name=host_name, address="10.8.8.8"))
|
||||
s.add(Host(id=hid2, name=quiet_host, address="10.8.8.9"))
|
||||
async with s.begin():
|
||||
for name, st, health, exit_code, proj, service, task in (
|
||||
(web, "running", "healthy", None, f"stack-{tag}", None, None),
|
||||
(db_, "running", "unhealthy", None, f"stack-{tag}", None, None),
|
||||
(job, "exited", None, 1, None, None, None),
|
||||
(f"{svc}.1.abc", "running", None, None, None, svc, f"abc-{tag}"),
|
||||
# The task a rolling update replaced: killed (137), not crashed.
|
||||
(replaced, "exited", None, 137, None, svc, f"old-{tag}"),
|
||||
):
|
||||
await s.execute(container_sql, {
|
||||
"h": hid, "n": name, "cid": uuid.uuid4().hex, "st": st, "rc": 0,
|
||||
"started": now - timedelta(hours=2), "now": now, "health": health,
|
||||
"exit": exit_code, "proj": proj, "svc": service, "task": task})
|
||||
for name in (job, replaced):
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_events (id, host_id, container_name, event, detail, at) "
|
||||
"VALUES (:id, :h, :n, 'die', 'exit 1', :at)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "n": name, "at": now - timedelta(minutes=10)})
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_events (id, host_id, container_name, event, detail, at) "
|
||||
"VALUES (:id, :h, :n, 'die', 'exit 1', :at)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "n": removed, "at": now - timedelta(minutes=12)})
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_logs (id, host_id, container_name, ts, stream, line) "
|
||||
"VALUES (:id, :h, :n, :ts, 'stderr', 'FATAL: cannot open database')"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "n": removed, "ts": now - timedelta(minutes=13)})
|
||||
for i, (stream, line) in enumerate((
|
||||
("stdout", "starting up"), ("stdout", "connected to db"),
|
||||
("stderr", "ERROR: 100% of disk_used"), ("stdout", "retrying"),
|
||||
("stdout", "x" * 5000))):
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_logs (id, host_id, container_name, ts, stream, line) "
|
||||
"VALUES (:id, :h, :n, :ts, :st, :line)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "n": web,
|
||||
"ts": now - timedelta(minutes=5 - i), "st": stream, "line": line})
|
||||
for i, cpu in enumerate((10.0, 30.0, 50.0)):
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_metrics (id, host_id, container_name, scraped_at, "
|
||||
"cpu_pct, mem_pct, mem_usage_bytes) VALUES (:id, :h, :n, :t, :c, 20.0, 1000)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "n": web,
|
||||
"t": now - timedelta(minutes=30 - i * 10), "c": cpu})
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_swarm_services (host_id, service_name, mode, desired, running, "
|
||||
"placement_json, tasks_json, updated_at) "
|
||||
"VALUES (:h, :n, 'replicated', 3, 1, '[]', :tasks, :now)"),
|
||||
{"h": hid, "n": svc, "now": now, "tasks": json.dumps([
|
||||
{"task_id": f"abc-{tag}", "slot": 1, "node_id": f"node-{tag}",
|
||||
"desired_state": "running", "state": "running", "message": "started",
|
||||
"error": "", "exit_code": None, "at": _ago(9)},
|
||||
{"task_id": f"bad-{tag}", "slot": 2, "node_id": f"node-{tag}",
|
||||
"desired_state": "shutdown", "state": "failed", "message": "started",
|
||||
"error": "task: non-zero exit (1)", "exit_code": 1, "at": _ago(10)},
|
||||
{"task_id": f"old-{tag}", "slot": 1, "node_id": f"node-{tag}",
|
||||
"desired_state": "shutdown", "state": "shutdown", "message": "shutdown",
|
||||
"error": "", "exit_code": 137, "at": _ago(11)}])})
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_swarm_nodes (host_id, node_id, hostname, role, availability, "
|
||||
"status, leader, updated_at) VALUES (:h, :nid, :hn, 'manager', 'active', 'ready', "
|
||||
"true, :now)"), {"h": hid, "nid": f"node-{tag}", "hn": host_name, "now": now})
|
||||
await s.execute(text(
|
||||
"INSERT INTO docker_disk_usage (host_id, layers_size, images_size, images_reclaimable, "
|
||||
"containers_size, volumes_size, build_cache_size, images_total, images_active, "
|
||||
"containers_count, volumes_count, updated_at) "
|
||||
"VALUES (:h, 1, 5000, 3000, 10, 20, 30, 4, 1, 4, 2, :now)"), {"h": hid, "now": now})
|
||||
await s.execute(text(
|
||||
"INSERT INTO host_agent_registrations (id, host_id, token_hash, token_created_at, "
|
||||
"agent_version, last_seen_at, created_at, updated_at) "
|
||||
"VALUES (:id, :h, :th, :t, '1.5.0', :seen, :t, :t)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid, "th": uuid.uuid4().hex + uuid.uuid4().hex,
|
||||
"t": now - timedelta(days=1), "seen": now - timedelta(seconds=20)})
|
||||
await s.execute(text(
|
||||
"INSERT INTO host_agent_registrations (id, host_id, token_hash, token_created_at, "
|
||||
"created_at, updated_at) VALUES (:id, :h, :th, :t, :t, :t)"),
|
||||
{"id": str(uuid.uuid4()), "h": hid2, "th": uuid.uuid4().hex + uuid.uuid4().hex,
|
||||
"t": now - timedelta(hours=2)})
|
||||
|
||||
async def _go():
|
||||
await _seed()
|
||||
_context._app_ctx.set(app)
|
||||
_context._caller_ctx.set(Caller(user_id="u", username="viewer", role=UserRole.viewer, token_id=0))
|
||||
|
||||
# ── list_containers: problems first, filters ───────────────────────
|
||||
mine = [c for c in (await dtools.list_containers(host=host_name, limit=500))["containers"]]
|
||||
names = [c["container_name"] for c in mine]
|
||||
assert set(names) >= {web, db_, job}
|
||||
problems = {c["container_name"]: c["problems"] for c in mine}
|
||||
assert problems[db_] == ["unhealthy"] and problems[job] == ["exited_nonzero"]
|
||||
assert problems[web] == []
|
||||
assert names.index(db_) < names.index(web) and names.index(job) < names.index(web)
|
||||
only = (await dtools.list_containers(host=hid, problem="unhealthy"))["containers"]
|
||||
assert [c["container_name"] for c in only] == [db_]
|
||||
proj = (await dtools.list_containers(compose_project=f"stack-{tag}"))["containers"]
|
||||
assert {c["container_name"] for c in proj} == {web, db_}
|
||||
with pytest.raises(ValueError, match="is not 'any' or one of"):
|
||||
await dtools.list_containers(problem="sad")
|
||||
|
||||
# ── get_container ──────────────────────────────────────────────────
|
||||
c = await dtools.get_container(web, range="1h")
|
||||
assert c["host_name"] == host_name and c["status"] == "running"
|
||||
assert c["cpu_pct_history"]["max"] == 50.0 and c["cpu_pct_history"]["points"] == 3
|
||||
crashed = await dtools.get_container(job, host=host_name)
|
||||
assert [e["event"] for e in crashed["events"]] == ["die"]
|
||||
# A unique part of a long swarm task name is enough.
|
||||
assert (await dtools.get_container(f"{svc}.1.ab"))["swarm_service"] == svc
|
||||
with pytest.raises(ValueError, match="list_containers"):
|
||||
await dtools.get_container(f"nope-{tag}")
|
||||
|
||||
# ── search_container_logs ──────────────────────────────────────────
|
||||
hit = await dtools.search_container_logs(container=web, text="100%", context=1)
|
||||
assert [m["line"] for m in hit["matches"]] == ["ERROR: 100% of disk_used"]
|
||||
assert hit["matches"][0]["before"] == ["connected to db"]
|
||||
assert hit["matches"][0]["after"] == ["retrying"]
|
||||
# % and _ match literally, not as wildcards.
|
||||
assert (await dtools.search_container_logs(container=web, text="o_n"))["matches"] == []
|
||||
page1 = await dtools.search_container_logs(container=web, limit=2)
|
||||
assert page1["truncated"] and len(page1["matches"]) == 2
|
||||
assert "characters omitted" in page1["matches"][0]["line"]
|
||||
assert len(page1["matches"][0]["line"]) < 2100
|
||||
page2 = await dtools.search_container_logs(container=web, limit=2, before=page1["next_before"])
|
||||
assert [m["line"] for m in page2["matches"]] == ["ERROR: 100% of disk_used", "connected to db"]
|
||||
errs = await dtools.search_container_logs(host=host_name, stream="stderr")
|
||||
# Host-wide search reaches a removed container's lines too.
|
||||
assert [m["container_name"] for m in errs["matches"]] == [web, removed]
|
||||
with pytest.raises(ValueError, match="context needs `container`"):
|
||||
await dtools.search_container_logs(text="x", context=2)
|
||||
|
||||
# ── a container Docker removed is still reachable by name ─────────
|
||||
gone = await dtools.get_container(removed)
|
||||
assert gone["removed"] is True and gone["host_name"] == host_name
|
||||
assert [e["event"] for e in gone["events"]] == ["die"] and "no longer exists" in gone["note"]
|
||||
assert (await dtools.get_container(web))["removed"] is False
|
||||
lines = await dtools.search_container_logs(container=removed, range="1h")
|
||||
assert lines["container_removed"] is True
|
||||
assert [m["line"] for m in lines["matches"]] == ["FATAL: cannot open database"]
|
||||
# A unique part of its name finds it through its events.
|
||||
assert (await dtools.get_container(f"2.gone{tag}"))["container_name"] == removed
|
||||
with pytest.raises(ValueError, match="may have been removed"):
|
||||
await dtools.get_container(f"nothing-{tag}")
|
||||
|
||||
# ── swarm, disk ────────────────────────────────────────────────────
|
||||
sw = await dtools.list_swarm_services(limit=500)
|
||||
s = next(x for x in sw["services"] if x["service_name"] == svc)
|
||||
assert (s["state"], s["running"], s["desired"]) == ("degraded", 1, 3)
|
||||
# Swarm's task records: the replaced task is history, the failed one counts.
|
||||
assert [r["container_name"] for r in s["replicas"]] == [f"{svc}.1.abc"]
|
||||
assert s["previous_tasks"] == 1 and s["tasks_known"] is True
|
||||
assert (s["failures_1h"], s["failures_24h"], s["crash_looping"]) == (1, 1, False)
|
||||
assert s["last_error"]["error"] == "task: non-zero exit (1)"
|
||||
assert s["last_error"]["node"] == host_name and s["last_error"]["age_s"] >= 600
|
||||
# Short of replicas since slot 2's task failed, ten minutes ago.
|
||||
assert 600 <= s["trouble_age_s"] < 900
|
||||
old = (await dtools.list_containers(host=host_name, status="exited", limit=500))["containers"]
|
||||
old = next(c for c in old if c["container_name"] == replaced)
|
||||
assert old["swarm_replaced"] is True and old["problems"] == []
|
||||
assert any(n["node_id"] == f"node-{tag}" and n["leader"] for n in sw["nodes"])
|
||||
disk = (await dtools.get_docker_disk_usage(host=host_name))["hosts"]
|
||||
assert disk[0]["images_reclaimable_bytes"] == 3000
|
||||
|
||||
# ── overview contributions ─────────────────────────────────────────
|
||||
async with app.db_sessionmaker() as db:
|
||||
findings = await dtools.mcp_overview(db)
|
||||
found = {(f["kind"], f["subject"]) for f in findings}
|
||||
agent_found = {(f["kind"], f["subject"]) for f in await atools.mcp_overview(db)}
|
||||
assert ("container_unhealthy", db_) in found
|
||||
assert ("container_exited_nonzero", job) in found
|
||||
assert ("swarm_service_degraded", svc) in found
|
||||
# Each says how long it has been wrong: the death, the failed task.
|
||||
since = {f["subject"]: f["age_s"] for f in findings}
|
||||
assert 600 <= since[job] < 900 and 600 <= since[svc] < 900
|
||||
assert not any(subject == replaced for _, subject in found)
|
||||
assert not any(subject == web for _, subject in found)
|
||||
assert ("agent_never_reported", quiet_host) in agent_found
|
||||
assert not any(subject == host_name for _, subject in agent_found)
|
||||
|
||||
# ── the web pages say the same ─────────────────────────────────────
|
||||
client = app.test_client()
|
||||
async with client.session_transaction() as sess:
|
||||
sess["user_id"], sess["user_role"], sess["username"] = "u", "viewer", "viewer"
|
||||
topo = await (await client.get("/plugins/docker/swarm")).get_data(as_text=True)
|
||||
assert svc in topo and "1 task failed in the last 24h" in topo
|
||||
assert "task: non-zero exit (1)" in topo
|
||||
cards = await (await client.get("/plugins/docker/rows")).get_data(as_text=True)
|
||||
assert f"{svc}.1.abc" in cards and replaced not in cards
|
||||
assert "task: non-zero exit (1)" in cards
|
||||
assert (await client.get("/plugins/docker/widget")).status_code == 200
|
||||
|
||||
# ── host agents ────────────────────────────────────────────────────
|
||||
agents = {a["host_name"]: a for a in (await atools.list_agents())["agents"]}
|
||||
assert agents[host_name]["state"] == "reporting" and agents[host_name]["agent_version"] == "1.5.0"
|
||||
assert agents[quiet_host]["state"] == "pending"
|
||||
|
||||
# ── registration through the plugin hook, gated on enablement ─────
|
||||
mcp = app.mcp_server
|
||||
mcp.register_plugin("docker", docker)
|
||||
mcp.register_plugin("host_agent", agent)
|
||||
assert {"list_containers", "search_container_logs", "list_agents"} <= set(mcp.tool_roles)
|
||||
assert mcp.tool_plugins["list_containers"] == "docker"
|
||||
assert set(mcp.overview_sources) >= {"docker", "host_agent"}
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -1,203 +0,0 @@
|
||||
"""Integration: the MCP diagnostic tools against real Postgres.
|
||||
|
||||
Seeds one small, uniquely named slice of a fleet — a host with a failing ping
|
||||
monitor, an agent reporting high CPU, a firing alert, an HTTPS monitor whose
|
||||
certificate is about to expire, a failed Ansible run, an active maintenance
|
||||
window and an audit event — then calls each tool as a viewer and checks that
|
||||
its slice comes back shaped as the tool promises. The database is shared with
|
||||
other tests, so assertions look for the seeded rows rather than exact totals.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
_NEEDS_DB = pytest.mark.skipif(
|
||||
not os.environ.get("STEWARD_DATABASE_URL"),
|
||||
reason="integration test needs a live Postgres (STEWARD_DATABASE_URL)",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_diagnostic_tools_over_a_seeded_fleet(app):
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.core.audit import log_audit
|
||||
from steward.mcp import _context
|
||||
from steward.mcp.tools import alerts, audit, hosts, metrics, monitors, overview
|
||||
from steward.models.alerts import (
|
||||
AlertEvent, AlertOperator, AlertRule, AlertState, AlertStateEnum, MaintenanceWindow,
|
||||
)
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.hosts import Host
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.models.monitors import Monitor, MonitorResult
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
uid, hid, rule_id = str(uuid.uuid4()), str(uuid.uuid4()), str(uuid.uuid4())
|
||||
ping_id, web_id, run_id = str(uuid.uuid4()), str(uuid.uuid4()), str(uuid.uuid4())
|
||||
host_name, username = f"mcp-host-{tag}", f"mcp-{tag}"
|
||||
|
||||
async def _seed():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
s.add(User(id=uid, username=username, email=f"{tag}@e.test",
|
||||
password_hash="x", role=UserRole.admin, is_active=True))
|
||||
s.add(Host(id=hid, name=host_name, address="10.9.9.9"))
|
||||
async with s.begin():
|
||||
s.add(Monitor(id=ping_id, name=f"ping-{tag}", type="icmp",
|
||||
target="10.9.9.9", host_id=hid, config_json="{}", enabled=True))
|
||||
s.add(Monitor(id=web_id, name=f"web-{tag}", type="http",
|
||||
target="https://bob:hunter2@web.test/health",
|
||||
config_json=json.dumps({"headers": {"Authorization": "Bearer s3cret"}}),
|
||||
enabled=True))
|
||||
async with s.begin():
|
||||
s.add(MonitorResult(monitor_id=ping_id, checked_at=now - timedelta(minutes=10),
|
||||
is_up=True, response_ms=1.0))
|
||||
s.add(MonitorResult(monitor_id=ping_id, checked_at=now - timedelta(minutes=5),
|
||||
is_up=False, error_msg="timeout"))
|
||||
s.add(MonitorResult(monitor_id=ping_id, checked_at=now - timedelta(minutes=1),
|
||||
is_up=False, error_msg="timeout"))
|
||||
s.add(MonitorResult(monitor_id=web_id, checked_at=now - timedelta(minutes=1),
|
||||
is_up=True, status_code=200, response_ms=40.0,
|
||||
tls_expires_at=now + timedelta(days=5)))
|
||||
for i, v in enumerate((10.0, 20.0, 95.0)):
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="cpu_pct", value=v,
|
||||
recorded_at=now - timedelta(minutes=40 - i * 20, seconds=30)))
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="mem_used_pct", value=50.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name + ":/",
|
||||
metric_name="disk_used_pct", value=41.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
s.add(AlertRule(id=rule_id, name=f"cpu hot {tag}", source_module="host_agent",
|
||||
resource_name=host_name, metric_name="cpu_pct",
|
||||
operator=AlertOperator.gt, threshold=90.0,
|
||||
consecutive_failures_required=1, enabled=True, created_by=uid))
|
||||
async with s.begin():
|
||||
s.add(AlertState(rule_id=rule_id, state=AlertStateEnum.firing,
|
||||
last_transition_at=now - timedelta(minutes=2)))
|
||||
s.add(AlertEvent(rule_id=rule_id, from_state="inactive", to_state="firing",
|
||||
metric_value=95.0, transitioned_at=now - timedelta(minutes=2)))
|
||||
s.add(MaintenanceWindow(name=f"patching {tag}", start_at=now - timedelta(hours=1),
|
||||
end_at=now + timedelta(hours=1), scope_source="host_agent",
|
||||
scope_resource=host_name, created_by=uid))
|
||||
s.add(AnsibleRun(id=run_id, playbook_path=f"site-{tag}.yml", source_name="t",
|
||||
status=AnsibleRunStatus.failed, started_at=now - timedelta(hours=1),
|
||||
finished_at=now - timedelta(minutes=55),
|
||||
results={"failures": ["web01: apt lock held"]}))
|
||||
await log_audit(app, uid, username, "host.created", "host", hid, {"name": host_name})
|
||||
|
||||
async def _go():
|
||||
await _seed()
|
||||
_context._app_ctx.set(app)
|
||||
_context._caller_ctx.set(Caller(user_id=uid, username=username,
|
||||
role=UserRole.viewer, token_id=0))
|
||||
|
||||
# ── get_overview ───────────────────────────────────────────────────
|
||||
ov = await overview.get_overview()
|
||||
assert ov["all_clear"] is False
|
||||
mine = {(f["kind"], f["subject"]): f for f in ov["findings"]
|
||||
if tag in f["subject"] or f["subject"] == host_name}
|
||||
down = mine[("monitor_down", f"ping-{tag}")]
|
||||
assert down["severity"] == "critical" and "timeout" in down["summary"]
|
||||
assert down["ids"]["monitor_id"] == ping_id
|
||||
# down_since is the first failure after the last success, not the latest check.
|
||||
assert 250 <= down["age_s"] <= 400
|
||||
assert mine[("tls_expiring", f"web-{tag}")]["severity"] == "warning"
|
||||
assert mine[("alert_firing", f"cpu hot {tag}")]["ids"]["rule_id"] == rule_id
|
||||
assert mine[("host_cpu_high", host_name)]["severity"] == "critical"
|
||||
assert "apt lock held" in mine[("ansible_run_failed", f"site-{tag}.yml")]["summary"]
|
||||
assert any(w["name"] == f"patching {tag}" for w in ov["active_maintenance"])
|
||||
sev = [f["severity"] for f in ov["findings"]]
|
||||
assert sev == sorted(sev, key=lambda s: {"critical": 0, "warning": 1}[s])
|
||||
|
||||
# ── hosts ──────────────────────────────────────────────────────────
|
||||
listed = {h["host_name"]: h for h in (await hosts.list_hosts(limit=500))["hosts"]}
|
||||
h = listed[host_name]
|
||||
assert h["status"] == "down" and h["agent"]["reporting"] is True
|
||||
assert h["agent"]["cpu_pct"] == 95.0
|
||||
|
||||
detail = await hosts.get_host(host_name.upper()) # names match case-insensitively
|
||||
assert detail["host_id"] == hid
|
||||
assert [m["state"] for m in detail["monitors"]] == ["down"]
|
||||
assert detail["monitors"][0]["error"] == "timeout"
|
||||
mount = detail["agent"]["mounts"]
|
||||
assert mount["items"][0]["mount"] == "/" and mount["items"][0]["disk_used_pct"] == 41.0
|
||||
assert mount["total"] == 1 and mount["truncated"] is False
|
||||
assert [a["rule_id"] for a in detail["active_alerts"]] == [rule_id]
|
||||
assert detail["ansible_target"] is None
|
||||
|
||||
series = await hosts.get_host_metrics(hid, "cpu_pct", "1h")
|
||||
assert series["max"] == 95.0 and series["points"] >= 1 and series["series"]
|
||||
listing = await hosts.get_host_metrics(host_name)
|
||||
assert {m["metric"] for m in listing["metrics"]} == {"cpu_pct", "mem_used_pct"}
|
||||
with pytest.raises(ValueError, match="no metric 'nope'"):
|
||||
await hosts.get_host_metrics(host_name, "nope")
|
||||
with pytest.raises(ValueError, match="list_hosts"):
|
||||
await hosts.get_host(f"no-such-{tag}")
|
||||
|
||||
# ── monitors ───────────────────────────────────────────────────────
|
||||
down_list = await monitors.list_monitors(state="down", limit=500)
|
||||
ping = next(m for m in down_list["monitors"] if m["monitor_id"] == ping_id)
|
||||
assert ping["host_name"] == host_name and ping["down_since"]
|
||||
web = await monitors.get_monitor(f"web-{tag}")
|
||||
assert "hunter2" not in web["target"] and "bob:***@" in web["target"]
|
||||
assert web["config"]["headers"] == {"Authorization": "<redacted>"}
|
||||
assert 4 <= web["tls_expires_in_days"] <= 5.1
|
||||
ping_detail = await monitors.get_monitor(ping_id, failures=5)
|
||||
assert ping_detail["heartbeat"] == "+--"
|
||||
assert [f["error"] for f in ping_detail["recent_failures"]] == ["timeout", "timeout"]
|
||||
with pytest.raises(ValueError, match="is not one of"):
|
||||
await monitors.list_monitors(state="broken")
|
||||
|
||||
# ── alerts ─────────────────────────────────────────────────────────
|
||||
firing = [a for a in (await alerts.list_alerts())["alerts"] if a["rule_id"] == rule_id]
|
||||
assert firing[0]["state"] == "firing" and firing[0]["condition"] == "cpu_pct > 90"
|
||||
hist = await alerts.get_alert_history(rule=f"cpu hot {tag}", range="24h")
|
||||
assert [(e["from_state"], e["to_state"]) for e in hist["events"]] == [("inactive", "firing")]
|
||||
rules = {r["rule_id"]: r for r in (await alerts.list_alert_rules(limit=500))["rules"]}
|
||||
assert rules[rule_id]["state"] == "firing"
|
||||
windows = (await alerts.list_maintenance_windows(limit=500))["windows"]
|
||||
assert any(w["name"] == f"patching {tag}" and w["active"] for w in windows)
|
||||
|
||||
# ── query_metrics, one argument at a time ──────────────────────────
|
||||
assert "host_agent" in (await metrics.query_metrics())["source_modules"]
|
||||
res = (await metrics.query_metrics("host_agent", limit=1000))["resources"]
|
||||
assert host_name in res and host_name + ":/" in res
|
||||
latest = await metrics.query_metrics("host_agent", host_name)
|
||||
assert {m["metric"] for m in latest["metrics"]} == {"cpu_pct", "mem_used_pct"}
|
||||
cpu = await metrics.query_metrics("host_agent", host_name, "cpu_pct", "1h")
|
||||
assert cpu["last"] == 95.0
|
||||
with pytest.raises(ValueError, match="no metrics recorded"):
|
||||
await metrics.query_metrics(f"nothing-{tag}")
|
||||
with pytest.raises(ValueError, match="is not one of"):
|
||||
await metrics.query_metrics("host_agent", host_name, "cpu_pct", "2w")
|
||||
|
||||
# ── audit ──────────────────────────────────────────────────────────
|
||||
events = (await audit.list_audit_events(action="host.", user=username))["events"]
|
||||
assert [(e["action"], e["entity_id"]) for e in events] == [("host.created", hid)]
|
||||
assert events[0]["detail"] == {"name": host_name}
|
||||
|
||||
# ── through the SDK: arguments validate and results serialise ──────
|
||||
await app.mcp_server.call_tool("list_hosts", {"limit": 5})
|
||||
await app.mcp_server.call_tool("get_overview", {})
|
||||
|
||||
asyncio.run(_go())
|
||||
@@ -1,62 +0,0 @@
|
||||
"""Drive /mcp through the app's ASGI callable, as a client's HTTP request would.
|
||||
|
||||
Quart's test_client never reaches /mcp (it is dispatched before Quart's
|
||||
pipeline), so MCP tests call app.asgi_app directly.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
|
||||
async def asgi_request(app, method, path, headers=None, body=None):
|
||||
raw = [(k.lower().encode(), v.encode()) for k, v in (headers or {}).items()]
|
||||
if body is not None:
|
||||
raw.append((b"content-length", str(len(body)).encode()))
|
||||
scope = {
|
||||
"type": "http", "asgi": {"version": "3.0", "spec_version": "2.3"},
|
||||
"http_version": "1.1", "method": method, "scheme": "http",
|
||||
"path": path, "raw_path": path.encode(), "query_string": b"",
|
||||
"root_path": "", "headers": raw,
|
||||
"client": ("testclient", 50000), "server": ("testserver", 80),
|
||||
}
|
||||
pending = [{"type": "http.request", "body": body or b"", "more_body": False}]
|
||||
sent = []
|
||||
responded = asyncio.Event()
|
||||
|
||||
# Like a real client, stay connected until the response is complete. An
|
||||
# SSE response listens for http.disconnect and cancels its stream on it,
|
||||
# so reporting a disconnect as soon as the body is read empties the reply.
|
||||
async def receive():
|
||||
if pending:
|
||||
return pending.pop(0)
|
||||
await responded.wait()
|
||||
return {"type": "http.disconnect"}
|
||||
|
||||
async def send(message):
|
||||
sent.append(message)
|
||||
if message["type"] == "http.response.body" and not message.get("more_body"):
|
||||
responded.set()
|
||||
|
||||
await asyncio.wait_for(app.asgi_app(scope, receive, send), timeout=15)
|
||||
start = next(m for m in sent if m["type"] == "http.response.start")
|
||||
data = b"".join(m.get("body", b"") for m in sent if m["type"] == "http.response.body")
|
||||
return start["status"], dict(start.get("headers", [])), data
|
||||
|
||||
|
||||
def rpc_result(data: bytes) -> dict:
|
||||
"""The JSON-RPC message from a JSON or a single-event SSE response body."""
|
||||
text = data.decode()
|
||||
if text.lstrip().startswith("{"):
|
||||
return json.loads(text)
|
||||
lines = [ln[5:].strip() for ln in text.splitlines() if ln.startswith("data:")]
|
||||
if not lines:
|
||||
raise AssertionError(f"no JSON-RPC message in response body: {text!r}")
|
||||
return json.loads(lines[-1])
|
||||
|
||||
|
||||
def rpc(method, params=None, id_=1) -> bytes:
|
||||
return json.dumps({"jsonrpc": "2.0", "id": id_, "method": method, "params": params or {}}).encode()
|
||||
|
||||
|
||||
HEADERS = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"}
|
||||
@@ -1,134 +0,0 @@
|
||||
"""The automation tools' pure parts and their gates: check mode by default,
|
||||
secrets kept out of what is stored, roles, and refusals that run nothing.
|
||||
|
||||
A real check-mode run, applied run and audit rows are covered against
|
||||
Postgres in tests/integration/test_mcp_automation.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from types import SimpleNamespace as NS
|
||||
|
||||
import pytest
|
||||
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.mcp.tools import automation
|
||||
from steward.models.users import UserRole
|
||||
|
||||
VARS = [
|
||||
{"name": "api_token", "default": "abc", "secret": True, "required": False, "prompt": None},
|
||||
{"name": "greeting", "default": "hi", "secret": False, "required": False, "prompt": None},
|
||||
{"name": "release", "default": "", "secret": False, "required": True, "prompt": "Release?"},
|
||||
]
|
||||
|
||||
VIEWER_TOOLS = {"list_playbooks", "get_playbook", "list_inventory", "list_ansible_runs",
|
||||
"get_ansible_run", "list_schedules", "wait_for_run"}
|
||||
OPERATOR_TOOLS = {"run_playbook", "cancel_ansible_run", "create_schedule", "update_schedule",
|
||||
"delete_schedule"}
|
||||
|
||||
|
||||
def test_split_vars_keeps_secrets_apart_from_what_is_stored():
|
||||
plain, secret = automation.split_vars(VARS, {"api_token": "s3cr3t", "greeting": "yo",
|
||||
"release": "1.2", "other": 5})
|
||||
assert plain == {"greeting": "yo", "release": "1.2", "other": 5}
|
||||
assert secret == {"api_token": "s3cr3t"}
|
||||
|
||||
|
||||
def test_split_vars_refuses_a_missing_prompt_and_odd_values():
|
||||
with pytest.raises(ValueError, match="prompts for release.*Nothing was run"):
|
||||
automation.split_vars(VARS, {"greeting": "yo"})
|
||||
with pytest.raises(ValueError, match="string, number or boolean"):
|
||||
automation.split_vars(VARS[:2], {"greeting": ["a"]})
|
||||
|
||||
|
||||
def test_public_variables_mask_secret_defaults():
|
||||
out = {v["name"]: v["default"] for v in automation.public_variables(VARS)}
|
||||
assert out == {"api_token": "<secret>", "greeting": "hi", "release": ""}
|
||||
|
||||
|
||||
def test_build_params():
|
||||
assert automation.build_params({}, "", "", True) == {"check": True}
|
||||
assert automation.build_params({"a": 1}, " web01 ", "deploy", False) == {
|
||||
"extra_vars_map": {"a": 1}, "limit": "web01", "tags": "deploy"}
|
||||
|
||||
|
||||
def test_writes_default_to_check_mode():
|
||||
for fn in (automation.run_playbook, automation.create_schedule, automation.set_alert_action):
|
||||
assert inspect.signature(fn).parameters["check"].default is True, fn.__name__
|
||||
|
||||
|
||||
async def test_run_playbook_will_not_apply_a_confirm_playbook_unconfirmed(monkeypatch):
|
||||
monkeypatch.setattr(automation, "_source", lambda name: {"name": name, "path": "/x"})
|
||||
monkeypatch.setattr(automation, "_playbook",
|
||||
lambda s, p: "# steward:confirm: true\n- hosts: all\n tasks: []\n")
|
||||
|
||||
def _no_run(*a, **k):
|
||||
raise AssertionError("a run was started")
|
||||
monkeypatch.setattr(automation, "trigger_run", _no_run)
|
||||
with pytest.raises(ValueError, match="flagged destructive.*confirmed=true"):
|
||||
await automation.run_playbook("s", "wipe.yml", check=False)
|
||||
|
||||
|
||||
def test_recap_totals_sum_over_hosts():
|
||||
results = {"hosts": {"a": {"ok": 3, "changed": 1}, "b": {"ok": 2, "failed": 1}}}
|
||||
assert automation.recap_totals(results) == {
|
||||
"hosts": 2, "ok": 5, "changed": 1, "failed": 1, "unreachable": 0, "skipped": 0}
|
||||
assert automation.recap_totals(None) is None
|
||||
|
||||
|
||||
def test_read_log_tail_and_grep(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(automation.executor, "artifact_path", lambda rid: str(tmp_path / "none.log"))
|
||||
run = NS(id="r1", output="\n".join(["PLAY [all]", "TASK [x]", "fatal: boom", "x" * 900, "PLAY RECAP"]))
|
||||
out = automation.read_log(run, tail=2, grep="FATAL")
|
||||
assert out["log_lines"] == 5
|
||||
assert out["tail"][1] == "PLAY RECAP" and out["tail"][0].endswith("[… 400 more]")
|
||||
assert out["matches"] == [{"line_no": 3, "line": "fatal: boom"}] and out["matches_total"] == 1
|
||||
(tmp_path / "r1.log").write_text("from the artifact\n")
|
||||
monkeypatch.setattr(automation.executor, "artifact_path", lambda rid: str(tmp_path / f"{rid}.log"))
|
||||
assert automation.read_log(run, tail=10, grep="")["tail"] == ["from the artifact"]
|
||||
|
||||
|
||||
def test_automation_tools_are_registered_with_their_roles(app):
|
||||
mcp = app.mcp_server
|
||||
for name in VIEWER_TOOLS:
|
||||
assert mcp.tool_roles[name] is UserRole.viewer, name
|
||||
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is True, name
|
||||
for name in OPERATOR_TOOLS:
|
||||
assert mcp.tool_roles[name] is UserRole.operator, name
|
||||
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is False, name
|
||||
assert mcp.tool_roles["set_alert_action"] is UserRole.admin
|
||||
for name in VIEWER_TOOLS | OPERATOR_TOOLS | {"set_alert_action"}:
|
||||
assert (mcp._tool_manager.get_tool(name).description or "").strip(), name
|
||||
|
||||
|
||||
def test_a_viewer_token_cannot_start_a_run(app):
|
||||
viewer = Caller(user_id="u", username="v", role=UserRole.viewer, token_id=1)
|
||||
operator = Caller(user_id="u", username="o", role=UserRole.operator, token_id=2)
|
||||
assert "needs the operator role" in app.mcp_server.refusal("run_playbook", viewer)
|
||||
assert app.mcp_server.refusal("run_playbook", operator) is None
|
||||
assert "needs the admin role" in app.mcp_server.refusal("set_alert_action", operator)
|
||||
|
||||
|
||||
def test_builtin_playbooks_get_steward_url_from_steward():
|
||||
from steward.mcp import _context
|
||||
app = NS(config={"PUBLIC_BASE_URL": ""})
|
||||
a, u = _context._app_ctx.set(app), _context._base_url_ctx.set("http://steward.lan")
|
||||
try:
|
||||
declared = [{"name": "steward_url", "default": "", "secret": False, "required": False,
|
||||
"prompt": None}]
|
||||
builtin = {"name": "steward-builtin"}
|
||||
got, filled = automation.steward_defaults(builtin, declared, {"agent_interval": 30})
|
||||
assert got == {"agent_interval": 30, "steward_url": "http://steward.lan"}
|
||||
assert filled == {"steward_url": "http://steward.lan"}
|
||||
# The configured public URL wins over the address the request came in on.
|
||||
app.config["PUBLIC_BASE_URL"] = "https://steward.example/"
|
||||
assert automation.steward_defaults(builtin, declared, {})[1] == {
|
||||
"steward_url": "https://steward.example"}
|
||||
# A value the caller passed stands; another source's playbook is untouched.
|
||||
assert automation.steward_defaults(builtin, declared, {"steward_url": "http://x"})[1] == {}
|
||||
assert automation.steward_defaults({"name": "homelab"}, declared, {}) == ({}, {})
|
||||
# A playbook that doesn't declare it gets nothing extra.
|
||||
assert automation.steward_defaults(builtin, [], {}) == ({}, {})
|
||||
finally:
|
||||
_context._app_ctx.reset(a)
|
||||
_context._base_url_ctx.reset(u)
|
||||
@@ -1,97 +0,0 @@
|
||||
"""The monitoring-config write tools: their validation refusals and their roles.
|
||||
|
||||
Monitor-config validation is covered in tests/core/test_monitors.py (the form
|
||||
and the tools share it). The writes themselves, their audit rows and the
|
||||
rename warning run against Postgres in
|
||||
tests/integration/test_mcp_monitoring_writes.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from steward.alerts import service as alert_svc
|
||||
from steward.mcp.tools import alerts
|
||||
from steward.models.alerts import AlertOperator
|
||||
from steward.models.users import UserRole
|
||||
|
||||
NOW = datetime(2026, 10, 6, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
OPERATOR_TOOLS = {"create_monitor", "update_monitor", "set_monitor_enabled", "create_alert_rule",
|
||||
"update_alert_rule", "acknowledge_alert", "create_maintenance_window",
|
||||
"end_maintenance_window"}
|
||||
ADMIN_TOOLS = {"delete_monitor", "delete_alert_rule"}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def known(monkeypatch):
|
||||
"""A source with one resource and its catalog metrics, without a database."""
|
||||
async def resources(db, src):
|
||||
return ["nas"] if src == "host_agent" else []
|
||||
monkeypatch.setattr(alert_svc, "resources_for", resources)
|
||||
|
||||
|
||||
async def _fields(**over):
|
||||
args = dict(name="cpu hot", source_module="host_agent", resource_name="nas",
|
||||
metric_name="cpu_pct", operator=">", threshold="90", consecutive_failures_required=2)
|
||||
args.update(over)
|
||||
return await alert_svc.rule_fields(None, **args)
|
||||
|
||||
|
||||
async def test_rule_fields_accepts_the_form_choices(known):
|
||||
f = await _fields()
|
||||
assert f["operator"] is AlertOperator.gt and f["threshold"] == 90.0
|
||||
assert f["consecutive_failures_required"] == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("over, why", [
|
||||
({"name": " "}, "needs a name"),
|
||||
({"source_module": "zabbix"}, "'zabbix' is not one of icmp, tcp"),
|
||||
({"resource_name": "web01"}, "host_agent has no resource 'web01'. Known: nas"),
|
||||
({"metric_name": "cpu"}, "does not report 'cpu'. Its metrics: cpu_pct"),
|
||||
({"operator": "=>"}, "'=>' is not one of >, <"),
|
||||
({"threshold": "high"}, "threshold must be a number"),
|
||||
({"consecutive_failures_required": 0}, "at least 1"),
|
||||
])
|
||||
async def test_rule_fields_refuses_with_a_reason(known, over, why):
|
||||
with pytest.raises(ValueError, match=why):
|
||||
await _fields(**over)
|
||||
|
||||
|
||||
def test_window_fields():
|
||||
f = alert_svc.window_fields(name="patching", start_at=NOW, end_at=NOW + timedelta(hours=1),
|
||||
scope_source="", scope_resource="nas")
|
||||
assert f["scope_source"] is None and f["scope_resource"] is None # resource needs a source
|
||||
with pytest.raises(ValueError, match="must end after it starts"):
|
||||
alert_svc.window_fields(name="x", start_at=NOW, end_at=NOW, scope_source=None,
|
||||
scope_resource=None)
|
||||
with pytest.raises(ValueError, match="'nope' is not one of"):
|
||||
alert_svc.window_fields(name="x", start_at=NOW, end_at=NOW + timedelta(minutes=1),
|
||||
scope_source="nope", scope_resource=None)
|
||||
|
||||
|
||||
async def test_maintenance_window_needs_exactly_one_end():
|
||||
with pytest.raises(ValueError, match="exactly one of duration_minutes or end_at"):
|
||||
await alerts.create_maintenance_window("x")
|
||||
with pytest.raises(ValueError, match="exactly one of duration_minutes or end_at"):
|
||||
await alerts.create_maintenance_window("x", duration_minutes=30, end_at="2026-10-06T13:00:00Z")
|
||||
with pytest.raises(ValueError, match="not an ISO-8601 time"):
|
||||
await alerts.create_maintenance_window("x", end_at="tonight")
|
||||
|
||||
|
||||
def test_when_reads_a_zoneless_time_as_utc():
|
||||
assert alerts._when("2026-10-06T22:00:00", "end_at") == datetime(2026, 10, 6, 22, tzinfo=timezone.utc)
|
||||
assert alerts._when("2026-10-06T22:00:00Z", "end_at").utcoffset() == timedelta(0)
|
||||
|
||||
|
||||
def test_write_tools_are_registered_with_their_roles(app):
|
||||
mcp = app.mcp_server
|
||||
for name in OPERATOR_TOOLS:
|
||||
assert mcp.tool_roles[name] is UserRole.operator, name
|
||||
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is False, name
|
||||
for name in ADMIN_TOOLS:
|
||||
assert mcp.tool_roles[name] is UserRole.admin, name
|
||||
assert mcp.tool_roles["get_alert_rule_options"] is UserRole.viewer
|
||||
for name in OPERATOR_TOOLS | ADMIN_TOOLS | {"get_alert_rule_options"}:
|
||||
assert (mcp._tool_manager.get_tool(name).description or "").strip(), name
|
||||
@@ -1,283 +0,0 @@
|
||||
"""The /mcp transport and the role-gated tool registry.
|
||||
|
||||
HTTP tests drive the app's ASGI callable directly: /mcp is dispatched before
|
||||
Quart's pipeline, so Quart's test_client never reaches it. The token store and
|
||||
the database probe are stood in for; tests/integration/test_mcp.py runs the
|
||||
same requests against real Postgres with a minted token.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from mcp.server.mcpserver.exceptions import ToolError
|
||||
from sqlalchemy.exc import OperationalError
|
||||
|
||||
from steward.core.api_tokens import Caller
|
||||
from steward.mcp import _context
|
||||
from steward.mcp.server import INSTRUCTIONS, StewardMCPServer
|
||||
from steward.models.users import UserRole
|
||||
from tests.mcp._http import HEADERS as _HEADERS, asgi_request, rpc, rpc_result
|
||||
|
||||
VIEWER = Caller(user_id="u1", username="ana", role=UserRole.viewer, token_id=1)
|
||||
OPERATOR = Caller(user_id="u1", username="ana", role=UserRole.operator, token_id=2)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def as_caller():
|
||||
# Each async test runs in its own task, i.e. a copy of the context, so a
|
||||
# caller set here can't leak into another test; resetting from the sync
|
||||
# teardown would fail ("created in a different Context").
|
||||
def _set(caller):
|
||||
_context._caller_ctx.set(caller)
|
||||
return _set
|
||||
|
||||
|
||||
def _server(plugins_cfg=None):
|
||||
app = MagicMock()
|
||||
app.config = {"PLUGINS": plugins_cfg or {}}
|
||||
return StewardMCPServer(app)
|
||||
|
||||
|
||||
# ── the registry ─────────────────────────────────────────────────────────────
|
||||
|
||||
async def test_every_registered_tool_declares_a_role(app):
|
||||
# Default-deny: the role map is the only door, so a tool the SDK knows about
|
||||
# without a role would be unreachable — and a sign someone bypassed
|
||||
# register_tool. Every tool in the built server must be in the map.
|
||||
mcp = app.mcp_server
|
||||
names = {t.name for t in mcp._tool_manager.list_tools()}
|
||||
assert names, "no MCP tools registered"
|
||||
assert names == set(mcp.tool_roles), f"tools without a declared role: {names - set(mcp.tool_roles)}"
|
||||
|
||||
|
||||
def test_a_tool_added_around_register_tool_is_unreachable():
|
||||
mcp = _server()
|
||||
|
||||
async def sneaky() -> dict:
|
||||
return {}
|
||||
mcp.add_tool(sneaky)
|
||||
assert "no Steward tool" in mcp.refusal("sneaky", OPERATOR)
|
||||
|
||||
|
||||
def test_register_tool_requires_a_role_enum_and_unique_names():
|
||||
mcp = _server()
|
||||
|
||||
async def t1() -> dict:
|
||||
return {}
|
||||
with pytest.raises(TypeError):
|
||||
mcp.register_tool(t1, role="viewer", read_only=True)
|
||||
mcp.register_tool(t1, role=UserRole.viewer, read_only=True)
|
||||
with pytest.raises(ValueError, match="already registered"):
|
||||
mcp.register_tool(t1, role=UserRole.viewer, read_only=True)
|
||||
|
||||
|
||||
def test_read_only_reaches_the_tool_annotations():
|
||||
mcp = _server()
|
||||
|
||||
async def look() -> dict:
|
||||
return {}
|
||||
|
||||
async def change() -> dict:
|
||||
return {}
|
||||
mcp.register_tool(look, role=UserRole.viewer, read_only=True)
|
||||
mcp.register_tool(change, role=UserRole.operator, read_only=False)
|
||||
assert mcp._tool_manager.get_tool("look").annotations.read_only_hint is True
|
||||
assert mcp._tool_manager.get_tool("change").annotations.read_only_hint is False
|
||||
assert mcp._tool_manager.get_tool("change").annotations.destructive_hint is True
|
||||
|
||||
|
||||
def test_instructions_fit_before_claude_codes_cut():
|
||||
# Claude Code keeps only the first ~2,048 characters of server instructions.
|
||||
assert len(INSTRUCTIONS) <= 1600
|
||||
|
||||
|
||||
# ── role and plugin gating ───────────────────────────────────────────────────
|
||||
|
||||
def _gated_server(plugins_cfg=None):
|
||||
mcp = _server(plugins_cfg)
|
||||
|
||||
async def look() -> dict:
|
||||
return {"ok": True}
|
||||
|
||||
async def change() -> dict:
|
||||
return {"changed": True}
|
||||
|
||||
async def plug() -> dict:
|
||||
return {}
|
||||
mcp.register_tool(look, role=UserRole.viewer, read_only=True)
|
||||
mcp.register_tool(change, role=UserRole.operator, read_only=False)
|
||||
mcp.register_tool(plug, role=UserRole.viewer, read_only=True, plugin="docker")
|
||||
return mcp
|
||||
|
||||
|
||||
async def test_list_tools_hides_what_the_role_cannot_reach(as_caller):
|
||||
mcp = _gated_server({"docker": {"enabled": True}})
|
||||
as_caller(VIEWER)
|
||||
assert {t.name for t in await mcp.list_tools()} == {"look", "plug"}
|
||||
as_caller(OPERATOR)
|
||||
assert {t.name for t in await mcp.list_tools()} == {"look", "change", "plug"}
|
||||
|
||||
|
||||
async def test_list_tools_is_empty_without_a_caller():
|
||||
assert await _gated_server().list_tools() == []
|
||||
|
||||
|
||||
async def test_call_above_role_is_refused_with_a_reason(as_caller):
|
||||
mcp = _gated_server()
|
||||
as_caller(VIEWER)
|
||||
with pytest.raises(ToolError, match="needs the operator role.*acts as viewer"):
|
||||
await mcp.call_tool("change", {})
|
||||
|
||||
|
||||
async def test_disabled_plugin_tools_are_hidden_and_refused(as_caller):
|
||||
mcp = _gated_server({"docker": {"enabled": False}})
|
||||
as_caller(OPERATOR)
|
||||
assert "plug" not in {t.name for t in await mcp.list_tools()}
|
||||
with pytest.raises(ToolError, match="docker plugin, which is disabled"):
|
||||
await mcp.call_tool("plug", {})
|
||||
|
||||
|
||||
async def test_undeclared_arguments_are_refused_not_dropped(as_caller):
|
||||
mcp = _server()
|
||||
|
||||
async def find(host: str = "") -> dict:
|
||||
return {"host": host}
|
||||
mcp.register_tool(find, role=UserRole.viewer, read_only=True)
|
||||
as_caller(VIEWER)
|
||||
with pytest.raises(ToolError, match=r"'hostname' \(did you mean 'host'\?\)"):
|
||||
await mcp.call_tool("find", {"hostname": "nas"})
|
||||
|
||||
|
||||
async def test_value_error_reaches_the_agent_as_its_reason(as_caller):
|
||||
mcp = _server()
|
||||
|
||||
async def picky() -> dict:
|
||||
raise ValueError("host 'nas' not found; list_hosts shows the names")
|
||||
mcp.register_tool(picky, role=UserRole.viewer, read_only=True)
|
||||
as_caller(VIEWER)
|
||||
with pytest.raises(ToolError, match="host 'nas' not found"):
|
||||
await mcp.call_tool("picky", {})
|
||||
|
||||
|
||||
async def test_database_failure_is_reported_as_an_outage(as_caller):
|
||||
mcp = _server()
|
||||
|
||||
async def reads() -> dict:
|
||||
raise OperationalError("SELECT 1", {}, ConnectionRefusedError())
|
||||
mcp.register_tool(reads, role=UserRole.viewer, read_only=True)
|
||||
as_caller(VIEWER)
|
||||
with pytest.raises(ToolError, match="can't reach its database"):
|
||||
await mcp.call_tool("reads", {})
|
||||
|
||||
|
||||
async def test_whoami_reports_the_effective_role(as_caller):
|
||||
from steward.mcp.tools.session import whoami
|
||||
as_caller(VIEWER)
|
||||
out = await whoami()
|
||||
assert out["user"] == "ana" and out["role"] == "viewer" and out["token_id"] == 1
|
||||
|
||||
|
||||
# ── plugin hook ──────────────────────────────────────────────────────────────
|
||||
|
||||
def test_plugin_tools_register_with_their_plugin_and_reload_cleanly():
|
||||
mcp = _server({"docker": {"enabled": True}})
|
||||
|
||||
async def list_containers() -> dict:
|
||||
return {}
|
||||
module = MagicMock()
|
||||
module.register_mcp_tools = lambda register: register(
|
||||
list_containers, role=UserRole.viewer, read_only=True)
|
||||
|
||||
mcp.register_plugin("docker", module)
|
||||
assert mcp.tool_plugins == {"list_containers": "docker"}
|
||||
mcp.register_plugin("docker", module) # hot-reload: no duplicate error
|
||||
assert mcp.tool_plugins == {"list_containers": "docker"}
|
||||
|
||||
|
||||
def test_a_raising_plugin_hook_leaves_the_server_intact():
|
||||
mcp = _server()
|
||||
module = MagicMock()
|
||||
module.register_mcp_tools = MagicMock(side_effect=RuntimeError("boom"))
|
||||
mcp.register_plugin("broken", module)
|
||||
assert mcp.tool_plugins == {}
|
||||
|
||||
|
||||
def test_a_plugin_hook_failing_halfway_registers_none_of_its_tools():
|
||||
mcp = _server()
|
||||
|
||||
async def first() -> dict:
|
||||
return {}
|
||||
|
||||
def hook(register):
|
||||
register(first, role=UserRole.viewer, read_only=True)
|
||||
register(first, role=UserRole.viewer, read_only=True) # duplicate → raises
|
||||
module = MagicMock()
|
||||
module.register_mcp_tools = hook
|
||||
mcp.register_plugin("half", module)
|
||||
assert mcp.tool_plugins == {}
|
||||
assert "first" not in mcp.tool_roles
|
||||
assert mcp._tool_manager.get_tool("first") is None
|
||||
|
||||
|
||||
def test_a_plugin_without_the_hook_contributes_nothing():
|
||||
mcp = _server()
|
||||
mcp.register_plugin("quiet", object())
|
||||
assert mcp.tool_plugins == {}
|
||||
|
||||
|
||||
# ── HTTP ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.fixture
|
||||
def db_up():
|
||||
with patch("steward.database.ensure_database_reachable", new=AsyncMock()):
|
||||
yield
|
||||
|
||||
|
||||
async def test_http_without_a_token_is_401(app, db_up):
|
||||
status, headers, body = await asgi_request(app, "POST", "/mcp", _HEADERS, rpc("tools/list"))
|
||||
assert status == 401
|
||||
assert headers[b"www-authenticate"].startswith(b"Bearer")
|
||||
assert b"AI access" in body
|
||||
|
||||
|
||||
async def test_http_with_an_unknown_token_is_401(app, db_up):
|
||||
with patch("steward.core.api_tokens.resolve_token", new=AsyncMock(return_value=None)):
|
||||
status, _, _ = await asgi_request(
|
||||
app, "POST", "/mcp", {**_HEADERS, "Authorization": "Bearer stw_nope"}, rpc("tools/list"))
|
||||
assert status == 401
|
||||
|
||||
|
||||
async def test_http_when_the_database_is_down_is_503(app):
|
||||
from steward.database import DatabaseUnavailable
|
||||
app._db_engine = MagicMock()
|
||||
with patch("steward.database.ensure_database_reachable",
|
||||
new=AsyncMock(side_effect=DatabaseUnavailable("connection refused"))):
|
||||
status, _, body = await asgi_request(
|
||||
app, "POST", "/mcp", {**_HEADERS, "Authorization": "Bearer stw_x"}, rpc("tools/list"))
|
||||
assert status == 503
|
||||
assert json.loads(body)["error"] == "database_unavailable"
|
||||
|
||||
|
||||
async def test_http_initialize_list_and_call_with_a_valid_token(app, db_up):
|
||||
auth = {**_HEADERS, "Authorization": "Bearer stw_valid"}
|
||||
with patch("steward.core.api_tokens.resolve_token", new=AsyncMock(return_value=VIEWER)):
|
||||
async with app.mcp_server.session_manager.run():
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc("initialize", {
|
||||
"protocolVersion": "2025-06-18", "capabilities": {},
|
||||
"clientInfo": {"name": "test", "version": "0"}}))
|
||||
assert status == 200, body
|
||||
assert rpc_result(body)["result"]["serverInfo"]["name"] == "steward"
|
||||
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc("tools/list", id_=2))
|
||||
assert status == 200, body
|
||||
names = {t["name"] for t in rpc_result(body)["result"]["tools"]}
|
||||
assert "whoami" in names
|
||||
|
||||
status, _, body = await asgi_request(app, "POST", "/mcp", auth, rpc(
|
||||
"tools/call", {"name": "whoami", "arguments": {}}, id_=3))
|
||||
assert status == 200, body
|
||||
result = rpc_result(body)["result"]
|
||||
assert not result.get("isError")
|
||||
assert json.loads(result["content"][0]["text"])["role"] == "viewer"
|
||||
@@ -1,268 +0,0 @@
|
||||
"""The diagnostic tools' pure parts: response shaping, lookup, redaction, ranking.
|
||||
|
||||
The queries behind each tool run against real Postgres in
|
||||
tests/integration/test_mcp_tools.py; these cover the logic that decides what
|
||||
an agent is told.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace as NS
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from steward.mcp.tools import _shape, overview
|
||||
from steward.mcp.tools.monitors import _config
|
||||
from steward.models.alerts import AlertOperator
|
||||
from steward.models.ansible import AnsibleRunStatus
|
||||
from steward.models.users import UserRole
|
||||
|
||||
NOW = datetime(2026, 10, 6, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
MCP4_TOOLS = {
|
||||
"get_overview", "list_hosts", "get_host", "get_host_metrics", "query_metrics",
|
||||
"list_monitors", "get_monitor", "list_alerts", "get_alert_history",
|
||||
"list_alert_rules", "list_maintenance_windows", "list_audit_events",
|
||||
}
|
||||
|
||||
|
||||
# ── shaping ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def test_iso_is_utc_with_z_and_naive_times_are_read_as_utc():
|
||||
assert _shape.iso(NOW) == "2026-10-06T12:00:00Z"
|
||||
assert _shape.iso(datetime(2026, 10, 6, 12, 0)) == "2026-10-06T12:00:00Z"
|
||||
assert _shape.iso(None) is None
|
||||
|
||||
|
||||
def test_age_s_never_negative():
|
||||
assert _shape.age_s(NOW - timedelta(seconds=90), NOW) == 90
|
||||
assert _shape.age_s(NOW + timedelta(seconds=5), NOW) == 0
|
||||
|
||||
|
||||
def test_clamp_limit():
|
||||
assert _shape.clamp_limit(0, 50, 500) == 50
|
||||
assert _shape.clamp_limit(-3, 50, 500) == 50
|
||||
assert _shape.clamp_limit(10, 50, 500) == 10
|
||||
assert _shape.clamp_limit(9999, 50, 500) == 500
|
||||
|
||||
|
||||
def test_bounded_says_when_and_how_it_cut():
|
||||
whole = _shape.bounded([1, 2], 5, key="xs")
|
||||
assert whole == {"xs": [1, 2], "total": 2, "truncated": False}
|
||||
cut = _shape.bounded(list(range(10)), 3, key="xs", more="Filter by `state`.")
|
||||
assert cut["xs"] == [0, 1, 2] and cut["total"] == 10 and cut["truncated"] is True
|
||||
assert cut["more"] == "Showing 3 of 10. Filter by `state`."
|
||||
|
||||
|
||||
def test_lookback_refuses_an_unknown_range_instead_of_guessing():
|
||||
since, key = _shape.lookback("7d")
|
||||
assert key == "7d"
|
||||
with pytest.raises(ValueError, match="'2w' is not one of"):
|
||||
_shape.lookback("2w")
|
||||
|
||||
|
||||
def test_summarize_and_downsample():
|
||||
pts = [(NOW + timedelta(minutes=i), float(i)) for i in range(1, 101)]
|
||||
s = _shape.summarize(pts, n=10)
|
||||
assert (s["points"], s["min"], s["max"], s["last"]) == (100, 1.0, 100.0, 100.0)
|
||||
assert s["avg"] == 50.5 and s["p95"] == 95.0
|
||||
assert len(s["series"]) == 10
|
||||
assert s["series"][0] == ["2026-10-06T12:01:00Z", 5.5]
|
||||
assert _shape.summarize([]) == {"points": 0, "series": []}
|
||||
|
||||
|
||||
def _rows(*names):
|
||||
return [NS(id=f"id-{i}", name=n) for i, n in enumerate(names)]
|
||||
|
||||
|
||||
def test_pick_by_id_then_case_insensitive_name():
|
||||
rows = _rows("nas", "web01")
|
||||
assert _shape.pick("id-1", rows, kind="host", list_tool="list_hosts").name == "web01"
|
||||
assert _shape.pick("NAS", rows, kind="host", list_tool="list_hosts").id == "id-0"
|
||||
|
||||
|
||||
def test_pick_explains_a_miss_and_an_ambiguity():
|
||||
with pytest.raises(ValueError, match=r"Did you mean 'web01'\?.*list_hosts"):
|
||||
_shape.pick("web1", _rows("nas", "web01"), kind="host", list_tool="list_hosts")
|
||||
with pytest.raises(ValueError, match="2 hosts are named 'nas'.*id-0, id-1"):
|
||||
_shape.pick("nas", _rows("nas", "NAS"), kind="host", list_tool="list_hosts")
|
||||
with pytest.raises(ValueError, match="pass a host name or id"):
|
||||
_shape.pick("", _rows("nas"), kind="host", list_tool="list_hosts")
|
||||
|
||||
|
||||
def test_redact_url_hides_only_the_password():
|
||||
assert _shape.redact_url("https://bob:pw@h.test:8443/x?q=1") == "https://bob:***@h.test:8443/x?q=1"
|
||||
assert _shape.redact_url("https://h.test/x") == "https://h.test/x"
|
||||
assert _shape.redact_url("10.0.0.1") == "10.0.0.1"
|
||||
|
||||
|
||||
def test_monitor_config_redacts_header_values():
|
||||
m = NS(config_json=json.dumps({"method": "GET", "headers": {"Authorization": "Bearer x"}}))
|
||||
assert _config(m) == {"method": "GET", "headers": {"Authorization": "<redacted>"}}
|
||||
assert _config(NS(config_json="not json")) == {}
|
||||
|
||||
|
||||
# ── overview findings ────────────────────────────────────────────────────────
|
||||
|
||||
def _monitor(mid, *, enabled=True, host_id=None):
|
||||
return NS(id=mid, name=f"mon-{mid}", type="http", target="https://x.test",
|
||||
enabled=enabled, host_id=host_id)
|
||||
|
||||
|
||||
def _result(*, up, error=None, tls_days=None):
|
||||
return NS(is_up=up, error_msg=error, checked_at=NOW - timedelta(minutes=1),
|
||||
tls_expires_at=NOW + timedelta(days=tls_days) if tls_days is not None else None)
|
||||
|
||||
|
||||
def test_monitor_findings():
|
||||
mons = [_monitor("a", host_id="h1"), _monitor("b"), _monitor("c"), _monitor("d", enabled=False),
|
||||
_monitor("e")]
|
||||
states = {
|
||||
"a": {"latest": _result(up=False, error="503"), "down_since": NOW - timedelta(hours=2)},
|
||||
"b": {"latest": _result(up=True, tls_days=2)},
|
||||
"c": {"latest": _result(up=True, tls_days=10)},
|
||||
"d": {"latest": _result(up=False)},
|
||||
"e": {"latest": _result(up=True, tls_days=60)},
|
||||
}
|
||||
out = {f["subject"]: f for f in overview.monitor_findings(mons, states, {"h1": "nas"}, NOW)}
|
||||
assert set(out) == {"mon-a", "mon-b", "mon-c"} # disabled and healthy left out
|
||||
down = out["mon-a"]
|
||||
assert down["severity"] == "critical" and down["kind"] == "monitor_down"
|
||||
assert "on nas" in down["summary"] and "503" in down["summary"]
|
||||
assert down["since"] == "2026-10-06T10:00:00Z"
|
||||
assert down["next"] == 'get_monitor(monitor="a")'
|
||||
assert (out["mon-b"]["severity"], out["mon-c"]["severity"]) == ("critical", "warning")
|
||||
# The TLS window comes from Settings → Thresholds.
|
||||
narrow = {f["subject"] for f in overview.monitor_findings(
|
||||
mons, states, {}, NOW, {"warn": 5, "crit": 1})}
|
||||
assert narrow == {"mon-a", "mon-b"}
|
||||
|
||||
|
||||
def test_alert_findings_severity_by_state():
|
||||
rule = NS(id="r1", name="cpu", metric_name="cpu_pct", operator=AlertOperator.gt,
|
||||
threshold=90.0, source_module="host_agent", resource_name="nas")
|
||||
rows = [(NS(state=NS(value=s), last_transition_at=NOW), rule)
|
||||
for s in ("firing", "pending", "acknowledged")]
|
||||
sev = {f["kind"]: f["severity"] for f in overview.alert_findings(rows)}
|
||||
assert sev == {"alert_firing": "critical", "alert_pending": "warning",
|
||||
"alert_acknowledged": "warning"}
|
||||
|
||||
|
||||
THRESHOLDS = {"cpu": {"warn": 80, "crit": 90}, "mem": {"warn": 80, "crit": 90},
|
||||
"disk": {"warn": 80, "crit": 90}}
|
||||
|
||||
|
||||
def test_agent_findings_use_the_configured_thresholds():
|
||||
hosts = [NS(id="h1", name="nas"), NS(id="h2", name="web"), NS(id="h3", name="db")]
|
||||
agent = {
|
||||
"nas": {"fresh": True, "cpu_pct": 95.0, "mem_used_pct": 85.0, "disk_used_pct_worst": 10.0},
|
||||
"web": {"fresh": False, "cpu_pct": 99.0, "last_at": NOW - timedelta(minutes=30)},
|
||||
}
|
||||
out = overview.agent_findings(hosts, agent, THRESHOLDS)
|
||||
got = {(f["subject"], f["kind"]): f["severity"] for f in out}
|
||||
assert got == {
|
||||
("nas", "host_cpu_high"): "critical",
|
||||
("nas", "host_mem_high"): "warning",
|
||||
# A stale agent's old numbers are not reported as current load.
|
||||
("web", "agent_stale"): "warning",
|
||||
}
|
||||
|
||||
|
||||
def test_run_findings_quote_the_failures():
|
||||
run = NS(id="r", playbook_path="site.yml", source_name="infra",
|
||||
status=AnsibleRunStatus.failed, started_at=NOW, finished_at=NOW,
|
||||
results={"failures": ["web01: apt lock held"]})
|
||||
f = overview.run_findings([run])[0]
|
||||
assert f["kind"] == "ansible_run_failed" and "apt lock held" in f["summary"]
|
||||
|
||||
|
||||
def test_rank_puts_critical_first_then_oldest():
|
||||
fs = [overview.finding("warning", "k", "w-old", "", since=NOW - timedelta(days=1)),
|
||||
overview.finding("critical", "k", "c-new", "", since=NOW - timedelta(minutes=1)),
|
||||
overview.finding("critical", "k", "c-old", "", since=NOW - timedelta(hours=1))]
|
||||
assert [f["subject"] for f in overview.rank(fs)] == ["c-old", "c-new", "w-old"]
|
||||
|
||||
|
||||
async def test_plugin_overview_sources_skip_disabled_plugins_and_survive_failures(monkeypatch):
|
||||
calls = []
|
||||
|
||||
async def good(db):
|
||||
calls.append("good")
|
||||
return [overview.finding("warning", "container_unhealthy", "redis", "unhealthy")]
|
||||
|
||||
async def broken(db):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
async def off(db):
|
||||
calls.append("off")
|
||||
return []
|
||||
|
||||
class _CM:
|
||||
async def __aenter__(self):
|
||||
return object()
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(overview, "db_session", lambda: _CM())
|
||||
server = MagicMock()
|
||||
server.overview_sources = {"docker": good, "snmp": broken, "unifi": off}
|
||||
server.plugin_enabled = lambda p: p != "unifi"
|
||||
found, unavailable = await overview._plugin_findings(server)
|
||||
assert [f["subject"] for f in found] == ["redis"]
|
||||
assert unavailable == ["snmp"] and calls == ["good"]
|
||||
|
||||
|
||||
def test_register_plugin_picks_up_an_overview_source():
|
||||
from steward.mcp.server import StewardMCPServer
|
||||
app = MagicMock()
|
||||
app.config = {"PLUGINS": {}}
|
||||
mcp = StewardMCPServer(app)
|
||||
|
||||
async def mcp_overview(db):
|
||||
return []
|
||||
mcp.register_plugin("docker", NS(mcp_overview=mcp_overview))
|
||||
assert mcp.overview_sources == {"docker": mcp_overview}
|
||||
mcp.register_plugin("docker", NS()) # reloaded without it
|
||||
assert mcp.overview_sources == {}
|
||||
|
||||
|
||||
# ── registration ─────────────────────────────────────────────────────────────
|
||||
|
||||
def test_diagnostic_tools_are_registered_read_only_for_viewers(app):
|
||||
mcp = app.mcp_server
|
||||
assert MCP4_TOOLS <= set(mcp.tool_roles)
|
||||
for name in MCP4_TOOLS:
|
||||
assert mcp.tool_roles[name] is UserRole.viewer, name
|
||||
assert mcp._tool_manager.get_tool(name).annotations.read_only_hint is True, name
|
||||
assert (mcp._tool_manager.get_tool(name).description or "").strip(), f"{name} has no docstring"
|
||||
|
||||
|
||||
def test_get_host_groups_parts_by_kind_and_summarises_cores():
|
||||
from steward.mcp.tools.hosts import agent_parts
|
||||
parts = {f"core{i}": {"cpu_pct": float(i)} for i in range(38)}
|
||||
parts.update({
|
||||
"/": {"disk_used_pct": 40.0, "disk_used_bytes": 4.0, "disk_total_bytes": 10.0},
|
||||
"/data": {"disk_used_pct": 91.0},
|
||||
"net:eth0": {"net_rx_bps": 10.0, "net_tx_bps": 5.0},
|
||||
"net:br-1": {"net_rx_bps": 900.0, "net_tx_bps": 0.0},
|
||||
"net:eth1": {"net_rx_bps": 0.0, "net_tx_bps": 0.0},
|
||||
"diskio:sda": {"disk_read_bps": 1.0, "disk_write_bps": 2.0},
|
||||
"temp:Package id 0": {"temp_c": 61.0},
|
||||
})
|
||||
got = agent_parts(parts, "box", limit=2)
|
||||
# Cores are a summary, so they can't crowd out the rest.
|
||||
assert got["cpu_cores"]["count"] == 38 and got["cpu_cores"]["max_pct"] == 37.0
|
||||
assert [c["core"] for c in got["cpu_cores"]["busiest"]] == [37, 36, 35]
|
||||
assert [m["mount"] for m in got["mounts"]["items"]] == ["/data", "/"] # fullest first
|
||||
# Busiest interface first; the cut is said, with how to get the rest.
|
||||
nets = got["interfaces"]
|
||||
assert [n["interface"] for n in nets["items"]] == ["br-1", "eth0"]
|
||||
assert nets["total"] == 3 and nets["truncated"] is True
|
||||
assert "parts_limit" in nets["more"] and 'resource="box:<part>"' in nets["more"]
|
||||
assert got["disk_io"]["items"][0]["device"] == "sda"
|
||||
assert got["temperatures"]["items"][0] == {"sensor": "Package id 0", "part": "temp:Package id 0",
|
||||
"temp_c": 61.0}
|
||||
assert "other" not in got
|
||||
@@ -65,35 +65,3 @@ def test_no_event_when_unchanged():
|
||||
old = {"web": {"status": "running", "health": "healthy", "oom_killed": False, "exit_code": None}}
|
||||
events = _derive_events(old, [_c("web", "running", health="healthy")])
|
||||
assert events == []
|
||||
|
||||
|
||||
# ── container-log row shaping (m79, pure) ─────────────────────────────────────
|
||||
|
||||
def test_log_rows_shapes_and_filters():
|
||||
from datetime import datetime, timezone
|
||||
from plugins.docker.ingest import _log_rows
|
||||
|
||||
rec_at = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
batches = [(rec_at, [
|
||||
{"container": "web", "stream": "stdout",
|
||||
"ts": "2026-01-01T00:00:01+00:00", "line": "hello"},
|
||||
{"container": "web", "stream": "stderr", "ts": None, "line": "no-ts"},
|
||||
{"container": "_steward", "stream": "stderr",
|
||||
"ts": None, "line": "truncated"}, # advisory marker → dropped
|
||||
{"container": "web", "stream": "weird", "ts": None, "line": "bad-stream"},
|
||||
{"container": "", "line": "x"}, # no name → dropped
|
||||
{"container": "web", "ts": None, "line": None}, # no line → dropped
|
||||
"not-a-dict", # malformed → dropped
|
||||
])]
|
||||
rows = list(_log_rows(batches, "host-1"))
|
||||
assert [r["line"] for r in rows] == ["hello", "no-ts", "bad-stream"]
|
||||
assert rows[0]["host_id"] == "host-1" and rows[0]["container_name"] == "web"
|
||||
assert rows[1]["ts"] == rec_at # None ts → recorded_at fallback
|
||||
assert rows[2]["stream"] == "stdout" # unknown stream normalised
|
||||
|
||||
|
||||
def test_log_rows_skips_non_list_records():
|
||||
from datetime import datetime, timezone
|
||||
from plugins.docker.ingest import _log_rows
|
||||
rec_at = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
assert list(_log_rows([(rec_at, None)], "h")) == []
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user