Compare commits
2
Commits
3cb2a95054
...
eea29123f9
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
eea29123f9 | ||
|
|
002181253e |
@@ -122,4 +122,5 @@ See `docs/plugins/` for a full plugin development guide.
|
||||
| `docs/plugins/overview.md` | Plugin system: how loading works, plugin.yaml schema, required exports |
|
||||
| `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide |
|
||||
| `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples |
|
||||
| `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe |
|
||||
| `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase |
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# Prometheus Plugin
|
||||
|
||||
The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric.
|
||||
|
||||
It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`).
|
||||
|
||||
---
|
||||
|
||||
## Adding a target
|
||||
|
||||
| Field | What it is |
|
||||
|---|---|
|
||||
| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. |
|
||||
| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. |
|
||||
| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. |
|
||||
| Scrape every | 10 s to an hour. Default 60 s. |
|
||||
| Check its HTTPS certificate | Off only for a self-signed certificate you trust. |
|
||||
| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. |
|
||||
| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. |
|
||||
|
||||
**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.
|
||||
|
||||
The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape.
|
||||
|
||||
---
|
||||
|
||||
## How series are stored
|
||||
|
||||
| On the endpoint | In Steward |
|
||||
|---|---|
|
||||
| A gauge or an untyped sample | Stored as it is, under its own name. |
|
||||
| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. |
|
||||
| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. |
|
||||
| NaN or ±Inf | Not stored. |
|
||||
|
||||
A series without labels is stored under the target's name. A labelled one is stored under `<target>:<label>=<value>,…`, in the order the endpoint gives the labels, for example `Archives:target=Steward,state=late`.
|
||||
|
||||
Each target also records four metrics about itself, under its own name:
|
||||
|
||||
| Metric | Meaning |
|
||||
|---|---|
|
||||
| `is_up` | 1 when the last scrape got metrics, 0 when it didn't. |
|
||||
| `scrape_ms` | How long the scrape took. |
|
||||
| `series_kept` | Series stored from the last scrape. |
|
||||
| `series_capped` | Series the last scrape left out because of the cap. |
|
||||
|
||||
A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.
|
||||
|
||||
Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).
|
||||
|
||||
---
|
||||
|
||||
## Alerting on a target
|
||||
|
||||
In **Alerts → New rule**, choose **Prometheus targets**, then the series (the resource) and the metric. Two rules worth having for any target:
|
||||
|
||||
- **It stopped answering:** resource `<target>`, metric `is_up`, `< 1`.
|
||||
- **Something was cut off:** resource `<target>`, metric `series_capped`, `> 0`.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Fabled Archives
|
||||
|
||||
Archives serves `/metrics` on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.
|
||||
|
||||
1. In Archives, **Settings → Metrics → Make a token**, named e.g. "Steward". Copy it; it is shown once.
|
||||
2. In Steward, **Prometheus → Add target**:
|
||||
- Name `Archives`
|
||||
- Metrics address `http://<archives host>:8081/metrics` (on a shared overlay network, the service name, e.g. `http://archives_app:8081/metrics`)
|
||||
- Bearer token: the one you copied
|
||||
- Keep: `fabledarchives_*`
|
||||
3. **Test**, then **Add target**.
|
||||
|
||||
Rules worth adding:
|
||||
|
||||
| Watch | Resource | Metric | Condition |
|
||||
|---|---|---|---|
|
||||
| A backup target is late | `Archives:target=<name>,target_id=<id>,state=late` | `fabledarchives_backup_target_state` | `>= 1` |
|
||||
| A backup target is missing | `Archives:target=<name>,target_id=<id>,state=missing` | `fabledarchives_backup_target_state` | `>= 1` |
|
||||
| A replication job is failing | `Archives:job=<name>,job_id=<id>,state=failing` | `fabledarchives_replication_job_state` | `>= 1` |
|
||||
| Archives' data disk is filling | `Archives` | `fabledarchives_data_disk_free_bytes` | `<` your floor |
|
||||
| Archives stopped answering | `Archives` | `is_up` | `< 1` |
|
||||
|
||||
The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.
|
||||
@@ -155,6 +155,9 @@ def create_app(
|
||||
for _name in ("metric_label", "metric_about", "metric_unit", "metric_value", "source_label",
|
||||
"cert_left", "field_label"):
|
||||
app.jinja_env.globals[_name] = getattr(_fmt, _name)
|
||||
# Which plugin settings fields are credentials (a password box, never echoed).
|
||||
from .core.settings import is_plugin_secret as _is_plugin_secret
|
||||
app.jinja_env.globals["is_plugin_secret"] = _is_plugin_secret
|
||||
from .alerts.service import rule_condition as _rule_condition
|
||||
app.jinja_env.globals["rule_condition"] = _rule_condition
|
||||
from .models.monitors import MONITOR_TYPE_LABELS as _type_labels
|
||||
|
||||
@@ -204,6 +204,11 @@ def _decode(value: Any, key: str = "") -> Any:
|
||||
exact setting that needs re-entering.
|
||||
"""
|
||||
from steward.core.crypto import decrypt_secret, is_encrypted
|
||||
if key.startswith(PLUGIN_PREFIX) and isinstance(value, dict):
|
||||
name = key[len(PLUGIN_PREFIX):]
|
||||
return {f: decrypt_secret(v, context=plugin_secret_key(name, f))
|
||||
if is_plugin_secret(f) and is_encrypted(v) else v
|
||||
for f, v in value.items()}
|
||||
return decrypt_secret(value, context=key) if is_encrypted(value) else value
|
||||
|
||||
|
||||
@@ -260,6 +265,38 @@ def source_token_key(name: str) -> str:
|
||||
return f"{SOURCES_KEY}[{name}].http_token"
|
||||
|
||||
|
||||
# A plugin's settings are one dict under plugin.<name>; a field whose name
|
||||
# says it holds a credential is sealed inside it, field by field. The same
|
||||
# test decides that the settings page shows it as a password box.
|
||||
PLUGIN_PREFIX = "plugin."
|
||||
_PLUGIN_SECRET_WORDS = ("password", "secret", "token")
|
||||
|
||||
|
||||
def is_plugin_secret(field: str) -> bool:
|
||||
"""Whether a plugin config field holds a credential: unifi's password,
|
||||
any *_secret or *_token."""
|
||||
return any(w in field for w in _PLUGIN_SECRET_WORDS)
|
||||
|
||||
|
||||
def seal_plugin_secrets(cfg: Any) -> tuple[Any, int]:
|
||||
"""Encrypt each plaintext secret field of one plugin's settings dict;
|
||||
returns (cfg, how many)."""
|
||||
from steward.core.crypto import encrypt_secret, is_encrypted
|
||||
if not isinstance(cfg, dict):
|
||||
return cfg, 0
|
||||
sealed, count = dict(cfg), 0
|
||||
for field, value in cfg.items():
|
||||
if is_plugin_secret(field) and isinstance(value, str) and value and not is_encrypted(value):
|
||||
sealed[field] = encrypt_secret(value)
|
||||
count += 1
|
||||
return sealed, count
|
||||
|
||||
|
||||
def plugin_secret_key(name: str, field: str) -> str:
|
||||
"""How a plugin's secret is named in logs and the undecryptable-secrets banner."""
|
||||
return f"{PLUGIN_PREFIX}{name}.{field}"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Async helpers (use inside request handlers / scheduled tasks)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
@@ -281,6 +318,8 @@ async def set_setting(session: AsyncSession, key: str, value: Any) -> None:
|
||||
if key in SECRET_KEYS and isinstance(value, str) and value:
|
||||
from steward.core.crypto import encrypt_secret
|
||||
to_store = encrypt_secret(value)
|
||||
elif key.startswith(PLUGIN_PREFIX):
|
||||
to_store, _ = seal_plugin_secrets(value)
|
||||
result = await session.execute(
|
||||
select(AppSetting).where(AppSetting.key == key)
|
||||
)
|
||||
@@ -569,6 +608,12 @@ def migrate_plaintext_secrets(db_url: str) -> int:
|
||||
if sealed:
|
||||
row.value_json = json.dumps(sources)
|
||||
converted += sealed
|
||||
for row in (await session.execute(select(AppSetting).where(
|
||||
AppSetting.key.startswith(PLUGIN_PREFIX)))).scalars():
|
||||
cfg, sealed = seal_plugin_secrets(json.loads(row.value_json))
|
||||
if sealed:
|
||||
row.value_json = json.dumps(cfg)
|
||||
converted += sealed
|
||||
finally:
|
||||
await engine.dispose()
|
||||
return converted
|
||||
@@ -605,6 +650,13 @@ def scan_undecryptable_secrets(db_url: str) -> list[str]:
|
||||
key = source_token_key(src.get("name", "?"))
|
||||
if _is_undecryptable(src.get("http_token"), key):
|
||||
bad.append(key)
|
||||
for row in (await session.execute(select(AppSetting).where(
|
||||
AppSetting.key.startswith(PLUGIN_PREFIX)))).scalars():
|
||||
cfg = json.loads(row.value_json)
|
||||
for field, value in (cfg.items() if isinstance(cfg, dict) else ()):
|
||||
key = plugin_secret_key(row.key[len(PLUGIN_PREFIX):], field)
|
||||
if is_plugin_secret(field) and _is_undecryptable(value, key):
|
||||
bad.append(key)
|
||||
finally:
|
||||
await engine.dispose()
|
||||
return bad
|
||||
|
||||
@@ -15,6 +15,7 @@ from steward.core.settings import (
|
||||
get_stored_plugin_names, find_orphaned_plugins,
|
||||
to_smtp_cfg, to_webhook_cfg, to_ansible_cfg, to_plugins_cfg,
|
||||
to_oidc_cfg, to_ldap_cfg, to_thresholds_cfg, to_access_cfg, DEFAULTS, seal_source_tokens,
|
||||
is_plugin_secret,
|
||||
)
|
||||
|
||||
settings_bp = Blueprint("settings", __name__, url_prefix="/settings")
|
||||
@@ -830,6 +831,12 @@ async def save_plugin_detail(name: str):
|
||||
old_enabled = to_plugins_cfg(old_settings).get(name, {}).get("enabled", False)
|
||||
|
||||
plugin_cfg = _build_plugin_cfg_from_form(plugin, form)
|
||||
# A secret is never sent back to the browser, so its box arrives blank
|
||||
# unless someone typed a new one: blank keeps the saved value.
|
||||
old_cfg = to_plugins_cfg(old_settings).get(name, {})
|
||||
for field, value in list(plugin_cfg.items()):
|
||||
if is_plugin_secret(field) and value == "" and old_cfg.get(field):
|
||||
plugin_cfg[field] = old_cfg[field]
|
||||
|
||||
async with current_app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
|
||||
@@ -108,10 +108,11 @@
|
||||
{% else %}
|
||||
<div class="form-group" style="margin:0;">
|
||||
<label style="font-size:0.82rem;">{{ field_label(cfg_key) }}</label>
|
||||
<input type="{% if 'password' in cfg_key or 'secret' in cfg_key %}password{% else %}text{% endif %}"
|
||||
{% set secret = is_plugin_secret(cfg_key) %}
|
||||
<input type="{% if secret %}password{% else %}text{% endif %}"
|
||||
name="plugin.{{ plugin._dir }}.{{ cfg_key }}"
|
||||
value="{% if 'password' in cfg_key or 'secret' in cfg_key %}{% else %}{{ plugin._config.get(cfg_key, cfg_default) }}{% endif %}"
|
||||
{% if 'password' in cfg_key or 'secret' in cfg_key %}placeholder="••••••••"{% endif %}>
|
||||
value="{% if secret %}{% else %}{{ plugin._config.get(cfg_key, cfg_default) }}{% endif %}"
|
||||
{% if secret %}autocomplete="new-password" placeholder="{{ 'kept; type to change' if plugin._config.get(cfg_key) else 'none saved' }}"{% endif %}>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
"""A plugin's credentials are sealed inside its settings dict (#5671): the
|
||||
UniFi password was stored in plain text, so a backup would have carried it."""
|
||||
from steward.core.crypto import init_crypto, is_encrypted
|
||||
from steward.core.settings import _decode, is_plugin_secret, seal_plugin_secrets
|
||||
|
||||
|
||||
def setup_function():
|
||||
init_crypto("unit-test-secret-key")
|
||||
|
||||
|
||||
def test_which_fields_are_credentials():
|
||||
assert is_plugin_secret("password") and is_plugin_secret("client_secret")
|
||||
assert is_plugin_secret("api_token")
|
||||
assert not is_plugin_secret("username") and not is_plugin_secret("host")
|
||||
|
||||
|
||||
def test_sealing_touches_only_plaintext_secrets_and_reading_opens_them():
|
||||
cfg = {"enabled": True, "host": "https://10.0.0.1", "username": "admin",
|
||||
"password": "hunter2", "poll_interval_seconds": 60}
|
||||
sealed, n = seal_plugin_secrets(cfg)
|
||||
assert n == 1 and is_encrypted(sealed["password"])
|
||||
assert {k: v for k, v in sealed.items() if k != "password"} == {
|
||||
k: v for k, v in cfg.items() if k != "password"}
|
||||
again, n2 = seal_plugin_secrets(sealed) # idempotent
|
||||
assert n2 == 0 and again == sealed
|
||||
assert _decode(sealed, "plugin.unifi") == cfg
|
||||
assert seal_plugin_secrets({"password": ""}) == ({"password": ""}, 0)
|
||||
assert seal_plugin_secrets("not a dict") == ("not a dict", 0)
|
||||
|
||||
|
||||
def test_only_plugin_settings_are_opened():
|
||||
# A dict under a non-plugin key is left alone: only plugin.* is opened.
|
||||
sealed, _ = seal_plugin_secrets({"password": "x"})
|
||||
assert _decode(sealed, "thresholds.cpu") == sealed
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Integration: a plugin's password is ciphertext in app_settings, reads back
|
||||
as itself, is sealed at startup if it was stored in plain text, and is kept
|
||||
when the settings form is saved with its box blank (#5671). Requires
|
||||
STEWARD_DATABASE_URL."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
if not os.environ.get("STEWARD_DATABASE_URL"):
|
||||
pytest.skip("needs Postgres")
|
||||
from steward.app import create_app
|
||||
return create_app(testing=False)
|
||||
|
||||
|
||||
def test_a_plugin_password_is_sealed_read_back_and_kept_on_a_blank_save(app):
|
||||
from sqlalchemy import delete, select
|
||||
|
||||
from steward.core.crypto import is_encrypted
|
||||
from steward.core.settings import get_setting, migrate_plaintext_secrets, set_setting
|
||||
from steward.models.settings import AppSetting
|
||||
from steward.models.users import User, UserRole
|
||||
|
||||
async def raw(key):
|
||||
async with app.db_sessionmaker() as s:
|
||||
row = (await s.execute(select(AppSetting).where(AppSetting.key == key))).scalar_one()
|
||||
return json.loads(row.value_json)
|
||||
|
||||
async def go():
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
cfg = {"enabled": False, "host": "https://10.9.9.1", "username": "admin",
|
||||
"password": f"pw-{tag}", "site": "default", "verify_ssl": False,
|
||||
"poll_interval_seconds": 60, "top_clients": 10}
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await set_setting(s, "plugin.unifi", cfg)
|
||||
stored = await raw("plugin.unifi")
|
||||
assert is_encrypted(stored["password"]) and stored["username"] == "admin"
|
||||
async with app.db_sessionmaker() as s:
|
||||
assert (await get_setting(s, "plugin.unifi"))["password"] == f"pw-{tag}"
|
||||
|
||||
# A row written before this fix, in plain text, is sealed at startup.
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
row = (await s.execute(select(AppSetting).where(AppSetting.key == "plugin.unifi"))).scalar_one()
|
||||
row.value_json = json.dumps(cfg)
|
||||
assert not is_encrypted((await raw("plugin.unifi"))["password"])
|
||||
await asyncio.to_thread(migrate_plaintext_secrets, app.config["DATABASE_URL"])
|
||||
assert is_encrypted((await raw("plugin.unifi"))["password"])
|
||||
|
||||
# Saving the settings page with the password box blank keeps it.
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
user = User(username=f"ps-{tag}", email=f"ps-{tag}@test", password_hash="x",
|
||||
role=UserRole.admin, is_active=True)
|
||||
s.add(user)
|
||||
uid = user.id
|
||||
client = app.test_client()
|
||||
async with client.session_transaction() as sess:
|
||||
sess.update(user_id=uid, user_role="admin", username=f"ps-{tag}",
|
||||
_csrf="test-token", login_at=time.time())
|
||||
resp = await client.post("/settings/plugins/unifi/", form={
|
||||
"csrf_token": "test-token", "plugin.unifi.host": "https://10.9.9.1",
|
||||
"plugin.unifi.username": "admin", "plugin.unifi.password": "",
|
||||
"plugin.unifi.site": "default", "plugin.unifi.poll_interval_seconds": "120",
|
||||
"plugin.unifi.top_clients": "10"})
|
||||
assert resp.status_code in (200, 302, 303), await resp.get_data(as_text=True)
|
||||
async with app.db_sessionmaker() as s:
|
||||
after = await get_setting(s, "plugin.unifi")
|
||||
assert after["password"] == f"pw-{tag}" and after["poll_interval_seconds"] == 120
|
||||
assert is_encrypted((await raw("plugin.unifi"))["password"])
|
||||
|
||||
page = await (await client.get("/settings/plugins/unifi/")).get_data(as_text=True)
|
||||
assert f"pw-{tag}" not in page and "kept; type to change" in page
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(delete(AppSetting).where(AppSetting.key == "plugin.unifi"))
|
||||
await s.execute(delete(User).where(User.id == uid))
|
||||
|
||||
asyncio.run(go())
|
||||
@@ -16,7 +16,7 @@ TEXT = """\
|
||||
# TYPE fa_backup_target_state gauge
|
||||
fa_backup_target_state{target="Steward",state="late"} 0
|
||||
# TYPE fa_requests_total counter
|
||||
fa_requests_total {count}
|
||||
fa_requests_total COUNT
|
||||
# TYPE noise gauge
|
||||
noise 1
|
||||
"""
|
||||
@@ -63,7 +63,7 @@ def test_a_scrape_records_series_rates_and_a_down_target_fires_its_rule(app):
|
||||
tokens.append(token)
|
||||
if "gone" in url:
|
||||
raise ScrapeError("Couldn't connect: connection refused.")
|
||||
return prometheus.parse(TEXT.format(count=next(counts)))
|
||||
return prometheus.parse(TEXT.replace("COUNT", str(next(counts))))
|
||||
|
||||
async def latest(name_prefix):
|
||||
async with app.db_sessionmaker() as s:
|
||||
|
||||
@@ -21,6 +21,7 @@ TOP_LEVEL = {
|
||||
"steward/templates/account/index.html",
|
||||
"plugins/docker/templates/docker/index.html",
|
||||
"plugins/snmp/templates/snmp/index.html",
|
||||
"plugins/prometheus/templates/prometheus/index.html",
|
||||
"plugins/traefik/templates/traefik/index.html",
|
||||
"plugins/unifi/templates/unifi/index.html",
|
||||
# Logged out, or no page to go back up from.
|
||||
|
||||
Reference in New Issue
Block a user