bvandeusenandClaude Opus 5.5
41d9d00216
Read from what Steward stored for Archives on 2026-10-14: the nightly Restic replication job had said "running" since the first scrape two days earlier, with no good copy yet and past both its late and missing times. Archives ranks "running" above lateness, so the widget called it healthy. - states blocks take warn_past / crit_past: a deadline metric per row (a *_timestamp_seconds). Once it has passed, a healthy row reads as a warning or a problem: "FullNightly-Restic is running, 11h 53m overdue". Quiet and already-failing rows keep their own reading. - The Archives template (version 2) uses the job's late_at / missing_at deadlines and shows its last good copy beside the next run. It says why section_up, the queue and the HTTP/audit counters are left out. - tests/plugins/prometheus/archives_live.py is the series as captured live, and test_archives_template.py renders the template over them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
69 lines
2.6 KiB
YAML
69 lines
2.6 KiB
YAML
# Fabled Archives, from its /metrics on the LAN-only backup listener (:8081).
|
|
# Written against the series Steward stores for Archives (#5693); see
|
|
# docs/plugins/prometheus.md.
|
|
#
|
|
# Left out on purpose:
|
|
# - fabledarchives_section_up: a section Archives can't read stops reporting
|
|
# its series, and the rows here that read them then say "not reported".
|
|
# - the background queue (fabledarchives_jobs, *_job_attempts_*), HTTP
|
|
# requests and the audit counters: they belong on charts and alert rules,
|
|
# not on a glance.
|
|
name: Fabled Archives
|
|
description: Backup targets, replication jobs, storage, integrity and the version running.
|
|
version: 2
|
|
expects: fabledarchives_
|
|
blocks:
|
|
- states:
|
|
title: Backup targets
|
|
metric: fabledarchives_backup_target_state
|
|
name: target
|
|
healthy: [fresh]
|
|
quiet: [disabled, waiting]
|
|
warning: [late]
|
|
columns:
|
|
- {label: Last backup, metric: fabledarchives_backup_target_last_backup_timestamp_seconds}
|
|
- {label: Stored, metric: fabledarchives_backup_target_stored_bytes}
|
|
- states:
|
|
title: Replication
|
|
metric: fabledarchives_replication_job_state
|
|
name: job
|
|
# Archives ranks "running" above lateness: a run that never ends still
|
|
# reads "running". Its own deadlines say when it is overdue.
|
|
healthy: [ok, running]
|
|
warning: [late, warning, never]
|
|
warn_past: fabledarchives_replication_job_late_at_timestamp_seconds
|
|
crit_past: fabledarchives_replication_job_missing_at_timestamp_seconds
|
|
columns:
|
|
- {label: Last good copy, metric: fabledarchives_replication_job_last_success_timestamp_seconds}
|
|
- {label: Next run, metric: fabledarchives_replication_job_next_run_timestamp_seconds}
|
|
- values:
|
|
title: Storage and integrity
|
|
items:
|
|
- label: Disk free
|
|
metric: fabledarchives_data_disk_free_bytes
|
|
of: fabledarchives_data_disk_size_bytes
|
|
warn_below: 10%
|
|
crit_below: 5%
|
|
sparkline: true
|
|
- label: Files checked
|
|
metric: fabledarchives_files_checked
|
|
of: fabledarchives_files
|
|
unit: count
|
|
- label: Malware held
|
|
metric: fabledarchives_malware_held_files
|
|
unit: count
|
|
warn_above: 0
|
|
- label: Notification channels failing
|
|
metric: fabledarchives_notification_channels_failing
|
|
unit: count
|
|
warn_above: 0
|
|
- table:
|
|
title: Stored per person
|
|
metric: fabledarchives_person_stored_bytes
|
|
rows: person
|
|
columns: kind
|
|
- info:
|
|
title: Running
|
|
metric: fabledarchives_build_info
|
|
labels: [version, channel, commit]
|