feat(backup): check a backup against what is in place, changing nothing (#5823)

#5823 "Check a backup: compare what it would restore with what is in
place, without restoring".

- steward/backup/check.py: pg_restore writes the backup's rows as COPY
  text with no database involved. The rows in place go out through COPY in
  the same form. Tables are compared on their primary keys, over the columns
  both sides have, and grouped by area.
- Sealed values are opened with each side's own key and compared as
  plaintext digests, so a secret sealed twice still matches. They are never
  shown. A backup whose key can't open its own secrets fails the check.
- What Steward records as it runs (last-seen times, a schedule's last run,
  backup history, scraped UniFi fields) is left out of the comparison:
  contents.ACTIVITY, STATE_SETTINGS and HAND_SET_ONLY.
- Settings → Backups:
  - a Check button on each sent backup, which fetches it from the bucket
    (new s3.get_file);
  - "Check a file" to upload one, with the size limit raised before the
    CSRF check reads the form;
  - a results page.
- Each backup Steward sends checks itself once written, and its history
  row says how that went.
- New CLI command `steward check-backup FILE`. CI's image smoke step checks a
  real backup against its source and expects a match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/templates/settings/backup_check.html

@@ -0,0 +1,74 @@
+{# settings/backup_check.html — a backup compared with what is in place (#5823):
+   the verdict, what the backup is (when, which Steward, its key), then each
+   area's count and the rows that differ. Nothing was changed to show it. #}
+{% extends "base.html" %}
+{% from "_macros.html" import crumbs %}
+{% from "settings/_tabs.html" import settings_tabs %}
+{% block title %}Settings — Backups — Check — Steward{% endblock %}
+{% block section_tabs %}{{ settings_tabs("backups") }}{% endblock %}
+{% block breadcrumb %}{{ crumbs([("Settings", "/settings/"), ("Backups", "/settings/backups/"), ("Check " ~ (name or "a backup"), "")]) }}{% endblock %}
+{% block content %}
+{% if error %}
+<p class="page-verdict"><span class="tone-crit">{{ name or "The backup" }} couldn't be checked. {{ error }}</span></p>
+{% else %}
+{% set tone = "tone-crit" if not report.ok else ("tone-ok" if not report.changes else "tone-warn") %}
+<p class="page-verdict"><span class="{{ tone }}">{{ report.summary() }}</span></p>
+
+<div class="card" style="max-width:640px;margin-bottom:1.25rem;">
+  <dl class="kv">
+    <dt>File</dt><dd>{{ report.name }}</dd>
+    <dt>Made</dt><dd>{% if report.made_at %}{{ report.made_at|when }}{% else %}—{% endif %}
+      by Steward {{ report.manifest.steward_version or "—" }}</dd>
+    <dt>Schema</dt><dd>
+      {%- if report.schema == "same" %}The same as this Steward's.
+      {%- elif report.schema == "older" %}<span class="tone-warn">Older than this Steward's.</span> A restore would replace everything, and Steward would bring it up to date on start.
+      {%- else %}<span class="tone-crit">Newer than this Steward's.</span> This Steward can't restore it; update Steward first.{% endif %}</dd>
+    <dt>Secret key</dt><dd>
+      {%- if report.key_same %}The one in use here.{% else %}<span class="tone-warn">A different one.</span> A restore would put it in place, and passwords and keys stored since would need entering again.{% endif %}
+      {%- if report.sealed %} It opens {{ report.sealed - report.unopened }} of the backup's {{ report.sealed }} sealed value{{ "s" if report.sealed != 1 }}.{% endif %}</dd>
+  </dl>
+</div>
+
+<div class="card-flush" style="margin-bottom:1.25rem;">
+  <table class="table">
+    <thead><tr><th>Area</th><th>Same</th><th>Differ</th><th>Only in the backup</th><th>Only here</th></tr></thead>
+    <tbody>
+      {% for a in report.areas %}
+      {% set differ = a.tables|sum(attribute="differ", start=[])|length %}
+      {% set only_b = a.tables|sum(attribute="only_backup", start=[])|length %}
+      {% set only_h = a.tables|sum(attribute="only_here", start=[])|length %}
+      <tr>
+        <td>{{ a.name }}</td>
+        <td>{{ a.same }}</td>
+        <td>{% if differ %}<span class="tone-warn">{{ differ }}</span>{% else %}—{% endif %}</td>
+        <td>{% if only_b %}<span class="tone-warn">{{ only_b }}</span>{% else %}—{% endif %}</td>
+        <td>{% if only_h %}<span class="tone-warn">{{ only_h }}</span>{% else %}—{% endif %}</td>
+      </tr>
+      {% endfor %}
+    </tbody>
+  </table>
+</div>
+
+{% for a in report.areas if a.changes or a.tables|selectattr("note")|list %}
+<h2 class="section-title" style="margin:1.5rem 0 0.75rem;">{{ a.name }}</h2>
+<ul class="check-list">
+  {% for t in a.tables %}
+  {% for d in t.differ %}
+  <li><strong>{{ d.label }}</strong> differs
+    {%- if d.secret %} (a sealed value; not shown).
+    {%- elif d.backup_value is not none or d.here_value is not none %}: the backup has <code>{{ d.backup_value if d.backup_value is not none else "nothing" }}</code>, here it is <code>{{ d.here_value if d.here_value is not none else "nothing" }}</code>.
+    {%- else %} in {{ d.columns|join(", ") }}.{% endif %}</li>
+  {% endfor %}
+  {% for label in t.only_backup %}<li><strong>{{ label }}</strong> is only in the backup: a restore brings it back.</li>{% endfor %}
+  {% for label in t.only_here %}<li><strong>{{ label }}</strong> is only here: a restore removes it.</li>{% endfor %}
+  {% if t.note %}<li class="tone-dim">{{ t.table }}: {{ t.note }}</li>{% endif %}
+  {% endfor %}
+</ul>
+{% endfor %}
+
+<p class="field-hint" style="margin-top:1rem;">Left out of the comparison: what Steward records as it runs
+  (when an agent or monitor was last seen, a schedule's last run, the backup history), since it changes by the
+  minute. A restore still brings those back as they were in the backup.</p>
+{% endif %}
+<p style="margin-top:1rem;"><a href="/settings/backups/">Back to Backups</a></p>
+{% endblock %}

steward/templates/settings/backups.html

@@ -1,6 +1,7 @@
 {# settings/backups.html — Steward's own backups (#5673): a verdict, the bucket
    they go to (Fabled Archives over S3) and the daily hour, then the recent
-   runs. Restoring is `steward restore`; see docs/core/backups.md. #}
+   runs, each with its own check and a Check button (#5823), and a file to
+   check. Restoring is `steward restore`; see docs/core/backups.md. #}
 {% extends "base.html" %}
 {% from "_macros.html" import crumbs %}
 {% from "settings/_tabs.html" import settings_tabs %}
@@ -115,17 +116,25 @@
 {% if history %}
 <div class="card-flush">
   <table class="table">
-    <thead><tr><th>When</th><th>Outcome</th><th>File</th><th>Size</th><th>Took</th><th>Started by</th></tr></thead>
+    <thead><tr><th>When</th><th>Outcome</th><th>File</th><th>Size</th><th>Took</th><th>Started by</th><th></th></tr></thead>
     <tbody>
       {% for r in history %}
       <tr>
         <td class="nowrap">{{ r.at|when if r.at else "—" }}</td>
-        <td>{% if r.ok %}<span class="tone-ok">Sent</span>{% else %}<span class="tone-crit">Failed</span>
+        <td>{% if r.ok %}<span class="tone-ok">Sent</span>
+          {%- if r.check %}<div class="cell-sub {{ 'tone-crit' if not r.check.ok else ('tone-warn' if r.check.changes else '') }}">
+            {%- if r.check.ok and not r.check.changes %}Checked: reads back whole{% else %}{{ r.check.summary }}{% endif %}</div>{% endif %}
+          {%- else %}<span class="tone-crit">Failed</span>
           <div class="cell-sub">{{ r.error }}</div>{% endif %}</td>
         <td class="cell-name"><span title="{{ r.name }}">{{ r.name or "—" }}</span></td>
         <td class="nowrap">{{ r.size|bytes if r.ok else "—" }}</td>
         <td class="nowrap">{{ r.secs|duration }}</td>
         <td>{{ "the schedule" if r.by == "schedule" else r.by }}</td>
+        <td>{% if r.ok and r.name %}
+          <form method="post" action="/settings/backups/check/">{{ csrf_field() }}
+            <input type="hidden" name="name" value="{{ r.name }}">
+            <button type="submit" class="btn btn-ghost btn-sm" title="Compare {{ r.name }} with what is in place; nothing changes">Check</button>
+          </form>{% endif %}</td>
       </tr>
       {% endfor %}
     </tbody>
@@ -134,6 +143,18 @@
 {% else %}
 <p class="tone-dim">None yet.</p>
 {% endif %}
-<p class="field-hint" style="margin-top:0.75rem;">To bring Steward back from one, see
-  <code>steward restore</code> in the backups guide (docs/core/backups.md).</p>
+<p class="field-hint" style="margin-top:0.75rem;">Check compares a backup with what is in place: what
+  a restore would bring back, change or remove, area by area. It changes nothing. To bring Steward back
+  from one, see <code>steward restore</code> in the backups guide (docs/core/backups.md).</p>
+
+<div class="card" style="max-width:640px;margin-top:1.25rem;">
+  <h2 class="section-title" style="margin-bottom:0.5rem;">Check a file</h2>
+  <form method="post" action="/settings/backups/check/file/" enctype="multipart/form-data"
+        style="display:flex;gap:0.75rem;flex-wrap:wrap;align-items:center;">{{ csrf_field() }}
+    <input type="file" name="file" accept=".tar" required>
+    <button type="submit" class="btn btn-ghost">Check</button>
+  </form>
+  <p class="field-hint" style="margin-top:0.5rem;">A <code>steward-&lt;time&gt;.tar</code> downloaded from
+    the bucket, or made with <code>steward backup</code>.</p>
+</div>
 {% endblock %}