feat(anomalies): a metric out of its normal is an anomaly to review (#5740)
#5740 "A metric that departs from its normal is recorded as an anomaly to review, and is findable after it ends", milestone 513. - Every minute each learned key's last 15 minutes is judged against its hour's normal: above the usual hold x1.25 and by the metric's floor opens an occurrence, which ends when the level comes back. - metric_anomalies holds one row per item to review; a repeat while it still waits joins it as an occurrence (metric_anomaly_spans). - An ended row nobody reviews within anomalies.review_days (14, Settings -> Thresholds) lapses; an ongoing one never does. - Learning skips every hour an occurrence covers unless its anomaly was marked normal: no normal is learned without review. - Alerts -> Anomalies lists those to review first, then the reviewed and lapsed under day dividers; marking is operator/admin and audited. - The Overview widget and get_overview warn of open ones and count those to review; the host page shows its own; the weekly report lists the week's and the lapsed; MCP list_anomalies and review_anomaly. - Reviewed and lapsed rows are pruned at data.retention_days, and kept in backups like alert history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -8,6 +8,7 @@ from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import delete
|
||||
|
||||
from steward.core.anomalies import prune as prune_anomalies
|
||||
from steward.models.alerts import AlertEvent
|
||||
from steward.models.ansible import AnsibleRun
|
||||
from steward.models.audit import AuditEvent
|
||||
@@ -76,6 +77,11 @@ async def run_cleanup(app: "Quart") -> dict:
|
||||
removed.append([f"{table}: older than {days} days", result.rowcount or 0])
|
||||
if result.rowcount:
|
||||
logger.info("Pruned %d rows from %s", result.rowcount, table)
|
||||
# Anomalies go by when their last occurrence ended, and only once
|
||||
# reviewed or lapsed: one still to review is never pruned (#5740).
|
||||
pruned = await prune_anomalies(session, cutoff)
|
||||
removed.append([f"metric_anomalies: reviewed, ended over {data_days} days ago",
|
||||
pruned])
|
||||
|
||||
files = await asyncio.to_thread(prune_run_logs, ARTIFACT_DIR, cutoff)
|
||||
removed.append([f"Ansible run log files: older than {data_days} days", files])
|
||||
|
||||
Reference in New Issue
Block a user