feat(anomalies): a metric out of its normal is an anomaly to review (#5740)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m24s
CI / unit (push) Successful in 1m37s
CI / publish (push) Skipped

#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.

- Every minute each learned key's last 15 minutes is judged against its
  hour's normal: above the usual hold x1.25 and by the metric's floor opens
  an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
  still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
  Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
  marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
  lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
  to review; the host page shows its own; the weekly report lists the
  week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
  in backups like alert history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 13:41:57 -04:00
co-authored by Claude Opus 5.5
parent cddf7fc6c4
commit 5569f9934e
34 changed files with 1343 additions and 25 deletions
+7 -3
View File
@@ -1,13 +1,13 @@
from __future__ import annotations
import asyncio
import uuid
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from quart import Blueprint, make_response, render_template, request, redirect, url_for, current_app, session
from sqlalchemy import select
from steward.ansible import executor, sources as ansible_src
from steward.alerts import service as alert_svc
from steward.auth.middleware import require_role
from steward.core import renames
from steward.core import anomalies as anom, renames
from steward.core.audit import audit_request, log_audit
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
from steward.models.hosts import Host
@@ -131,10 +131,14 @@ async def host_detail(host_id: str):
linkable_targets = (await db.execute(
select(AnsibleTarget).where(AnsibleTarget.host_id.is_(None))
.order_by(AnsibleTarget.name))).scalars().all()
# The host's metric anomalies, its disks and interfaces included (#5740).
anomalies = await anom.views(db, await anom.list_anomalies(
db, source="host_agent", resource=host.name,
since=datetime.now(timezone.utc) - timedelta(days=30), limit=10), links=False)
return await render_template(
"hosts/detail.html",
host=host, monitors=monitors, uptime=uptime, alerts=alerts,
host=host, monitors=monitors, uptime=uptime, alerts=alerts, anomalies=anomalies,
linked_target=linked_target, linkable_targets=linkable_targets,
ansible_sources=_ansible_source_names(),
)