feat(crypto): fail loudly on unpersistable secret key; flag undecryptable secrets

Follow-up to the incident where an unwritable /data made Steward silently mint a
fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only
discovered when an Ansible run failed. Make both failure modes loud and visible.

- config._resolve_secret_key: if a new key must be generated but can't be
  persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and
  refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data
  writable by uid 1000). Also raises a clear error if an existing key file can't
  be read. First-run on a writable volume still generates + persists normally.
- core.settings: detect secrets stored as ciphertext that won't decrypt with the
  current key (scan_undecryptable_secrets at startup; _is_undecryptable helper).
  Cached in memory; set_setting discards a key on a fresh write so the banner
  clears without a restart.
- app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject
  undecryptable_secrets into the template context.
- base.html: admin banner naming which secrets to re-enter, each linked to its
  settings tab.
- compose.deploy.yml: document the uid-1000 /data write requirement and the
  STEWARD_SECRET_KEY option for multi-node Swarm.
- Tests: secret-key resolver (reuse existing file, generate when writable, raise
  when unpersistable) and the undecryptable-secret detection helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

steward/core/settings.py

@@ -150,6 +150,34 @@ def _decode(value: Any, key: str = "") -> Any:
     return decrypt_secret(value, context=key) if is_encrypted(value) else value
 
 
+# Secret settings that are stored as ciphertext but won't decrypt with the
+# current app key (key rotated or lost). Surfaced as an admin banner so the
+# operator knows precisely which secrets to re-enter — instead of finding out
+# only when something that uses one fails. Refreshed at startup
+# (scan_undecryptable_secrets) and kept live: re-entering a secret clears it
+# without a restart (set_setting discards it on a fresh write).
+_undecryptable_secrets: set[str] = set()
+
+
+def get_undecryptable_secrets() -> list[str]:
+    """Sorted keys whose stored ciphertext won't decrypt (for the UI banner)."""
+    return sorted(_undecryptable_secrets)
+
+
+def _is_undecryptable(stored: Any, key: str = "") -> bool:
+    """True iff `stored` is an encrypted token that fails to decrypt.
+
+    A failed decrypt returns the ciphertext unchanged (still enc-prefixed), so a
+    value that is still encrypted after a decrypt attempt is undecryptable.
+    """
+    from steward.core.crypto import decrypt_secret, is_encrypted
+    return (
+        isinstance(stored, str)
+        and is_encrypted(stored)
+        and is_encrypted(decrypt_secret(stored, context=key))
+    )
+
+
 # ─────────────────────────────────────────────────────────────────────────────
 # Async helpers (use inside request handlers / scheduled tasks)
 # ─────────────────────────────────────────────────────────────────────────────
@@ -182,6 +210,12 @@ async def set_setting(session: AsyncSession, key: str, value: Any) -> None:
         row.value_json = json.dumps(to_store)
         row.updated_at = now
 
+    # A fresh write of a secret is encrypted with the current key (or cleared to
+    # plaintext), so it's decryptable now — drop any stale "undecryptable" flag
+    # so the banner clears without a restart.
+    if key in SECRET_KEYS:
+        _undecryptable_secrets.discard(key)
+
 
 async def get_all_settings(session: AsyncSession) -> dict[str, Any]:
     """Return flat key→value dict with defaults filled in for missing keys."""
@@ -359,6 +393,37 @@ def migrate_plaintext_secrets(db_url: str) -> int:
     return asyncio.run(_run())
 
 
+def scan_undecryptable_secrets(db_url: str) -> list[str]:
+    """Populate the undecryptable-secrets cache from the DB; return the keys found.
+
+    Run once at startup, AFTER migrate_plaintext_secrets and init_crypto: any row
+    that's encrypted but won't decrypt with the current key was sealed under a
+    different (lost/rotated) key and must be re-entered. Surfaced via the admin
+    banner (get_undecryptable_secrets).
+    """
+    async def _run() -> list[str]:
+        from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker
+        engine = create_async_engine(db_url, echo=False)
+        factory = async_sessionmaker(engine, expire_on_commit=False)
+        bad: list[str] = []
+        try:
+            async with factory() as session:
+                result = await session.execute(
+                    select(AppSetting).where(AppSetting.key.in_(SECRET_KEYS))
+                )
+                for row in result.scalars():
+                    if _is_undecryptable(json.loads(row.value_json), row.key):
+                        bad.append(row.key)
+        finally:
+            await engine.dispose()
+        return bad
+
+    global _undecryptable_secrets
+    found = asyncio.run(_run())
+    _undecryptable_secrets = set(found)
+    return sorted(found)
+
+
 # ─────────────────────────────────────────────────────────────────────────────
 # External URL helper
 # ─────────────────────────────────────────────────────────────────────────────