feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,7 +15,7 @@ from steward.core.settings import (
|
||||
get_stored_plugin_names, find_orphaned_plugins,
|
||||
to_smtp_cfg, to_webhook_cfg, to_ansible_cfg, to_plugins_cfg,
|
||||
to_oidc_cfg, to_ldap_cfg, to_thresholds_cfg, to_access_cfg, DEFAULTS, seal_source_tokens,
|
||||
is_plugin_secret,
|
||||
is_plugin_secret, NORM_FLOORS,
|
||||
)
|
||||
|
||||
settings_bp = Blueprint("settings", __name__, url_prefix="/settings")
|
||||
@@ -145,6 +145,8 @@ _THRESHOLD_FIELDS = [
|
||||
("error_5xx_pct", "traefik.threshold.error_5xx_pct", True),
|
||||
("error_4xx_pct", "traefik.threshold.error_4xx_pct", True),
|
||||
("min_requests", "traefik.threshold.min_requests", False),
|
||||
# Each normal's floor (#5742), named by its own setting key.
|
||||
*[(f"norms.floor.{name}", f"norms.floor.{name}", True) for name in NORM_FLOORS],
|
||||
]
|
||||
|
||||
# Docker time-series retention windows (days). Read fresh by the cleanup task,
|
||||
@@ -165,7 +167,8 @@ async def thresholds():
|
||||
settings = await get_all_settings(db)
|
||||
from steward.core.plugin_manager import _LOADED_PLUGINS
|
||||
return await render_template("settings/thresholds.html", settings=settings,
|
||||
traefik_on="traefik" in _LOADED_PLUGINS)
|
||||
traefik_on="traefik" in _LOADED_PLUGINS,
|
||||
plugins_on=set(_LOADED_PLUGINS))
|
||||
|
||||
|
||||
@settings_bp.post("/thresholds/")
|
||||
|
||||
Reference in New Issue
Block a user