fix(6244): the instance keeps its own SECRET_KEY in /data, and docker-compose.yml is the compose users deploy from
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m9s
CI & Build / Python tests (push) Successful in 2m7s
CI & Build / Build & push image (push) Successful in 30s

With no SECRET_KEY set, Grimoire fell back to a fixed placeholder. The
quickstart compose substituted change-me-in-production, so the README's
own update step (pull, up -d) recreated the container on that placeholder,
and every credential sealed under the key the user started with stopped
opening.

- config: with neither SECRET_KEY nor SECRET_KEY_FILE, the first start
  writes a random key to DATA_DIR/secret_key (mode 0600, published with a
  hard link so concurrent starts agree), and later starts read it back.
  An explicit key wins and leaves the file alone. A missing
  SECRET_KEY_FILE or an empty key file stops the start instead of
  switching keys. With no data directory (a run outside the container)
  the process gets a temporary key and says so.
- app: warns on a public placeholder key, on a temporary key, and when
  /data is not a mounted volume.
- Dockerfile creates /data.
- docker-compose.yml is now the deploy compose (published image, both
  volumes, nothing to set). The build-from-source compose is
  docker-compose.dev.yml, and docker-compose.quickstart.yml is gone. The
  Swarm stack file gains the /data volume and keeps its explicit key,
  because a Swarm can move the app away from a node-local volume.
- README: a Deploy section with the compose inline, SECRET_KEY optional,
  "keep both volumes", and updates that keep the key. A test holds the
  README's copy equal to the file.
- Installs that ran on the change-me-in-production placeholder get a key
  of their own on the new compose; Settings then asks once for saved
  forge tokens and the backup secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-22 16:13:43 -04:00
co-authored by Claude Opus 5.5
parent 81fe851509
commit 787685ac63
15 changed files with 329 additions and 75 deletions
+3 -1
View File
@@ -4,4 +4,6 @@ POSTGRES_USER=fabled
POSTGRES_PASSWORD=fabled
POSTGRES_DB=grimoire
# PGDATA_VOLUME=grimoire_pgdata
SECRET_KEY=dev-secret-change-me
# Grimoire keeps its own SECRET_KEY in the /data volume. Set one here only to
# use a key you manage yourself; it then wins over the kept one.
# SECRET_KEY=
+4
View File
@@ -56,5 +56,9 @@ ENV APP_BUILD_KEY=$BUILD_KEY
ENV APP_CHANNEL=$BUILD_CHANNEL
ENV APP_COMMIT=$BUILD_COMMIT
# The instance's own SECRET_KEY, the model cache and diagnostics live here.
# Mount a volume on it so they outlive the container.
RUN mkdir -p /data
EXPOSE 5000
CMD ["sh", "-c", "alembic upgrade head && hypercorn 'grimoire.app:create_app()' --bind 0.0.0.0:5000 --keep-alive 600"]
+5 -5
View File
@@ -3,22 +3,22 @@
# --- Docker ---
build:
docker compose build
docker compose -f docker-compose.dev.yml build
up:
docker compose up -d
docker compose -f docker-compose.dev.yml up -d
down:
docker compose down
docker compose -f docker-compose.dev.yml down
logs:
docker compose logs -f
docker compose -f docker-compose.dev.yml logs -f
health:
curl -s http://localhost:5000/api/health
migrate:
docker compose exec app alembic upgrade head
docker compose -f docker-compose.dev.yml exec app alembic upgrade head
# --- Local dev (requires: uv sync --extra dev and cd frontend && npm ci) ---
+59 -19
View File
@@ -8,17 +8,57 @@
Claude records what it learned, decided and built in Grimoire, and that record reaches the next session, so a solution is recalled instead of reinvented. Notes, tasks, plans, rules, lessons and reusable code live in one place. Claude reaches them through a built-in MCP endpoint and the bundled Claude Code plugin, and people read and edit the same records in the web UI. There is no language model inside Grimoire: Claude is the only assistant.
## Quick start
## Deploy
You need Docker with Compose. Download [`docker-compose.quickstart.yml`](docker-compose.quickstart.yml), then:
You need Docker with Compose. Save this as `docker-compose.yml` (it is the [`docker-compose.yml`](docker-compose.yml) in this repository), then run `docker compose up -d` beside it:
```bash
SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(32))') \
docker compose -f docker-compose.quickstart.yml up -d
```yaml
# Fabled Grimoire: the app and its database.
# docker compose up -d
# Then open http://localhost:5000. The first account registered becomes admin.
#
# Keep both volumes. app_data holds the instance's own SECRET_KEY, which seals
# saved credentials; pgdata holds everything else.
services:
app:
image: git.fabledsword.com/bvandeusen/fabledgrimoire:latest
ports:
- "5000:5000"
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
# Behind an HTTPS reverse proxy:
# BASE_URL: "https://grimoire.example.com"
# SECURE_COOKIES: "true"
# TRUST_PROXY_HEADERS: "true"
volumes:
- app_data:/data
depends_on:
- db
restart: unless-stopped
db:
# Postgres 17 with the vector extension Grimoire needs.
image: pgvector/pgvector:pg17
stop_grace_period: 120s
volumes:
- pgdata:/var/lib/postgresql/data
environment:
POSTGRES_USER: ${POSTGRES_USER:-grimoire}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-grimoire}
POSTGRES_DB: ${POSTGRES_DB:-grimoire}
restart: unless-stopped
volumes:
app_data:
pgdata:
```
Open `http://localhost:5000`. The first account you register becomes the admin.
Nothing has to be set first. On its first start Grimoire makes its own `SECRET_KEY` and keeps it in the `app_data` volume, so restarts, reboots and updates all use the same key.
To connect Claude, create an API key under **Settings → MCP Access**, then in Claude Code:
```
@@ -30,34 +70,34 @@ The plugin asks for your Grimoire URL and the API key. Settings → MCP Access s
## Configuration
Set these in the environment or a `.env` file beside the compose file:
Everything is optional. Set these in the environment or in a `.env` file beside the compose file:
| Variable | Needed | What it does |
|----------|--------|--------------|
| `SECRET_KEY` | Yes | Signs sessions and seals stored credentials. Set it once and keep it |
| Variable | When | What it does |
|----------|------|--------------|
| `BASE_URL` | Behind a proxy | The public URL, used in email links and SSO redirects |
| `SECURE_COOKIES` | Over HTTPS | `true` when Grimoire is served over HTTPS |
| `TRUST_PROXY_HEADERS` | Behind a proxy | `true` to read the client address from the proxy's headers |
| `POSTGRES_PASSWORD` | Recommended | The database password (defaults to `grimoire`) |
| `POSTGRES_PASSWORD` | Before the first start | The database password (defaults to `grimoire`; the database is reachable only inside the compose network). Postgres takes it once, when the volume is created |
| `SECRET_KEY` | Rarely | A key you manage yourself, instead of the one Grimoire keeps. It wins over the kept key; remove it to go back |
Everything else (registration, email, SSO, git forge connections, retrieval tuning, backups) is in **Settings**, or listed in [Configuration](docs/configuration.md).
## Setup pitfalls
- **Keep your `SECRET_KEY`.** Forge tokens and the backup bucket's secret are sealed under it. If it changes, Settings asks you to enter those credentials again.
- **The embedding model downloads on first use**, into the `/data` volume. Without internet access, semantic search falls back to keyword search until the model is in place.
- **Keep both volumes.** `pgdata` holds everything Grimoire stores. `app_data` holds the instance's key, and forge tokens and the backup bucket's secret are sealed under it. If the key is lost, Settings asks you to enter those credentials again; nothing else is lost.
- **The embedding model downloads on first use**, into the `app_data` volume. Without internet access, semantic search falls back to keyword search until the model is in place.
- **Behind a reverse proxy, turn off response buffering** for `/mcp`, because MCP responses stream. [Configuration](docs/configuration.md#behind-a-reverse-proxy) has an Nginx example.
## Updating and backups
```bash
docker compose -f docker-compose.quickstart.yml pull
docker compose -f docker-compose.quickstart.yml up -d
docker compose pull
docker compose up -d
```
Database migrations run when the app starts. Claude Code updates the plugin from its marketplace; run `/reload-plugins` after an update.
Database migrations run when the app starts, and the key stays in its volume. Claude Code updates the plugin from its marketplace; run `/reload-plugins` after an update.
Everything Grimoire stores is in the database. **Settings → Data → Scheduled backups** writes a daily backup to a folder or an S3-compatible bucket, and `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](docs/backups.md).
**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](docs/backups.md).
## Features
@@ -92,7 +132,8 @@ The full list is in [Features](docs/features.md).
| What you see | Why | Fix |
|--------------|-----|-----|
| `SECRET_KEY is set to the insecure default` at startup | `SECURE_COOKIES=true` while `SECRET_KEY` is still the built-in default | Set `SECRET_KEY` |
| `SECRET_KEY is set to the public placeholder` at startup | `SECURE_COOKIES=true` while `SECRET_KEY` is `dev-secret-change-me` | Remove `SECRET_KEY`, or set a real one |
| Settings asks for forge tokens or the backup secret again | The instance's key changed: the `app_data` volume was replaced, or `SECRET_KEY` was set or removed | Enter them again, or restore the previous key |
| `Can't locate revision identified by '…'` at startup | An older image is running against a database a newer one has already migrated | Run the newer image; migrations only move forward |
| The plugin's tools are missing in Claude Code | The plugin isn't installed, or Claude Code hasn't reloaded it | `/plugin` to install or update it, then `/reload-plugins` |
| Search finds only exact words | The embedding model hasn't downloaded yet | Give the app internet access once, or wait for the first download to finish |
@@ -100,8 +141,7 @@ The full list is in [Features](docs/features.md).
## Development
```bash
cp .env.example .env # then set SECRET_KEY
docker compose up --build
docker compose -f docker-compose.dev.yml up --build
```
[Development](docs/development.md) covers checks, migrations, the plugin and CI. [Architecture](docs/architecture.md) describes how it is built, and the [API reference](docs/api-reference.md) lists every REST endpoint.
@@ -1,36 +1,35 @@
# Fabled Grimoire from the pre-built image:
# docker compose -f docker-compose.quickstart.yml up -d
# Then open http://localhost:5000 (the first account registered becomes admin)
# and connect Claude from Settings → MCP Access.
# Grimoire built from this checkout, for development:
# docker compose -f docker-compose.dev.yml up --build
services:
app:
image: git.fabledsword.com/bvandeusen/fabledgrimoire:latest
build: .
ports:
- "5000:5000"
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
# Set SECRET_KEY in the environment or a .env file beside this one.
SECRET_KEY: "${SECRET_KEY:-change-me-in-production}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
volumes:
- app_data:/data
depends_on:
- db
restart: unless-stopped
volumes:
- app_data:/data
# Or a bind mount, for direct host access to the app's data:
# - ./data:/data
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
# Optional: a SearXNG instance to list as the web-search backend in Integrations.
# SEARXNG_URL: "http://searxng:8080"
db:
# Postgres 17 with the vector extension Grimoire needs.
image: pgvector/pgvector:pg17
stop_grace_period: 120s
restart: unless-stopped
volumes:
- pgdata:/var/lib/postgresql/data
environment:
POSTGRES_USER: ${POSTGRES_USER:-grimoire}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-grimoire}
POSTGRES_DB: ${POSTGRES_DB:-grimoire}
restart: unless-stopped
volumes:
app_data:
pgdata:
app_data:
+6
View File
@@ -3,10 +3,15 @@ services:
image: git.fabledsword.com/bvandeusen/fabledgrimoire:latest
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${DB_PASSWORD}@db:5432/${POSTGRES_DB:-grimoire}"
# A Swarm can start the app on another node, away from the key kept in
# a node-local /data volume, so a stack sets its key itself (or mounts
# a Docker secret and sets SECRET_KEY_FILE to its path).
SECRET_KEY: "${SECRET_KEY}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
TRUST_PROXY_HEADERS: "true"
SECURE_COOKIES: "true"
volumes:
- app_data:/data
networks:
- backend
@@ -26,6 +31,7 @@ services:
volumes:
pgdata:
name: ${PGDATA_VOLUME:-grimoire_pgdata}
app_data:
networks:
backend:
+19 -12
View File
@@ -1,33 +1,40 @@
# Fabled Grimoire: the app and its database.
# docker compose up -d
# Then open http://localhost:5000. The first account registered becomes admin.
#
# Keep both volumes. app_data holds the instance's own SECRET_KEY, which seals
# saved credentials; pgdata holds everything else.
services:
app:
build: .
image: git.fabledsword.com/bvandeusen/fabledgrimoire:latest
ports:
- "5000:5000"
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
# Behind an HTTPS reverse proxy:
# BASE_URL: "https://grimoire.example.com"
# SECURE_COOKIES: "true"
# TRUST_PROXY_HEADERS: "true"
volumes:
- app_data:/data
depends_on:
- db
restart: unless-stopped
volumes:
- app_data:/data
# Or a bind mount, for direct host access to the app's data:
# - ./data:/data
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
SECRET_KEY: "${SECRET_KEY:-dev-secret-change-me}"
# Optional: a SearXNG instance to list as the web-search backend in Integrations.
# SEARXNG_URL: "http://searxng:8080"
db:
# Postgres 17 with the vector extension Grimoire needs.
image: pgvector/pgvector:pg17
stop_grace_period: 120s
restart: unless-stopped
volumes:
- pgdata:/var/lib/postgresql/data
environment:
POSTGRES_USER: ${POSTGRES_USER:-grimoire}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-grimoire}
POSTGRES_DB: ${POSTGRES_DB:-grimoire}
restart: unless-stopped
volumes:
pgdata:
app_data:
pgdata:
+1 -1
View File
@@ -74,7 +74,7 @@ Every database wait on startup has a deadline. Migrations wait for the database
- **OIDC** single sign-on with PKCE, optionally replacing local logins ([SSO / OAuth](sso-oauth.md)).
- **Rate limits** on register, login and forgot-password, per client IP (`TRUST_PROXY_HEADERS` reads the IP from the proxy's headers).
Credentials Grimoire stores for outside services (forge tokens, the backup destination's S3 secret) are sealed under a key derived from `SECRET_KEY`.
Credentials Grimoire stores for outside services (forge tokens, the backup destination's S3 secret) are sealed under a key derived from `SECRET_KEY`, which the instance makes on first start and keeps in `/data/secret_key` unless one is set.
## Repository layout
+2 -1
View File
@@ -31,7 +31,8 @@ differently. It does **not** hold:
- **Credentials:** API keys and forge connections. After a restore, issue new
API keys and add forge connections again. (Forge tokens are also sealed under
`SECRET_KEY` in the database itself, so a copy of the database doesn't carry
them in readable form either.)
them in readable form either. The key lives in `/data/secret_key`, outside
the database.)
- **Logs and notifications.**
- **The backup destination's own S3 secret.**
+12 -11
View File
@@ -11,7 +11,8 @@ Any variable marked *secret* can also be read from a file: set `<NAME>_FILE` to
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` *(secret)* | `postgresql+asyncpg://grimoire:grimoire@localhost:5432/grimoire` | PostgreSQL connection string, using the `asyncpg` driver |
| `SECRET_KEY` *(secret)* | `dev-secret-change-me` | Signs sessions and seals stored credentials. Set it before first use and keep it (see below) |
| `SECRET_KEY` *(secret)* | kept in `DATA_DIR` | Signs sessions and seals stored credentials. Unset, the instance makes its own on first start and keeps it in `DATA_DIR/secret_key`. Set it only to manage the key yourself; it wins over the kept one (see below) |
| `DATA_DIR` | `/data` | Where the instance keeps its own `SECRET_KEY`. Mount a volume here |
| `BASE_URL` | `http://localhost:5000` | Public URL, used in email links and OIDC redirects |
| `SECURE_COOKIES` | `false` | `true` when Grimoire is served over HTTPS |
| `TRUST_PROXY_HEADERS` | `false` | `true` behind a reverse proxy, to read the client IP from `X-Forwarded-For` / `X-Real-IP` |
@@ -56,27 +57,27 @@ Admins can set SMTP in **Settings**; these are the defaults until they do.
## Running it
### Quick start
### Docker Compose
[`docker-compose.quickstart.yml`](../docker-compose.quickstart.yml) runs the published image with its own database:
[`docker-compose.yml`](../docker-compose.yml) runs the published image with its own database, and needs nothing set:
```bash
SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(32))') \
docker compose -f docker-compose.quickstart.yml up -d
docker compose up -d
```
Open `http://localhost:5000`. The first account registered becomes the admin; after that, registration stays closed until an admin opens it in Settings or sends an invite.
Update with `docker compose pull && docker compose up -d`. The key and the data stay in their volumes.
### From source
```bash
cp .env.example .env # then set SECRET_KEY
docker compose up --build
docker compose -f docker-compose.dev.yml up --build
```
### Docker Swarm
[`docker-compose.prod.yml`](../docker-compose.prod.yml) is a stack file: the app and database on a private overlay network, with `SECURE_COOKIES` and `TRUST_PROXY_HEADERS` on. Provide `DB_PASSWORD` and `SECRET_KEY` in the environment:
[`docker-compose.prod.yml`](../docker-compose.prod.yml) is a stack file: the app and database on a private overlay network, with `SECURE_COOKIES` and `TRUST_PROXY_HEADERS` on. A Swarm can move the app to another node, away from a node-local `/data` volume, so the stack sets its own key: provide `DB_PASSWORD` and `SECRET_KEY` in the environment (or mount a Docker secret and set `SECRET_KEY_FILE`):
```bash
DB_PASSWORD=... SECRET_KEY=... docker stack deploy -c docker-compose.prod.yml grimoire
@@ -89,9 +90,9 @@ The shipped files leave out healthchecks and restart policies on purpose. The ap
### Volumes
- **Database** (`pgdata`): everything Grimoire stores.
- **App data** (`/data`): the embedding model cache and diagnostics. Losing it costs a re-download, nothing else.
- **App data** (`/data`): the instance's own `SECRET_KEY`, the embedding model cache and diagnostics. Losing it costs the key: Settings then asks for saved forge tokens and the backup secret again, and the model downloads again.
Back up the database with **Settings → Data → Scheduled backups** ([Backups](backups.md)).
Back up the database with **Settings → Data → Scheduled backups** ([Backups](backups.md)). A database backup doesn't carry the key; to move sealed credentials to a new instance, copy `/data/secret_key` with it.
## Behind a reverse proxy
@@ -115,7 +116,7 @@ With `TRUST_PROXY_HEADERS=true`, make sure the proxy replaces any `X-Forwarded-F
## Security checklist
- **Set a strong `SECRET_KEY`, and keep it.** Forge tokens and the backup destination's S3 secret are sealed under a key derived from it, so a copy of the database alone can't read them. If the key changes (say, a restore onto an instance with a different key), Settings marks those credentials and asks for them again; nothing else breaks. The app logs a warning at startup while the default key is in use, and refuses to start with it when `SECURE_COOKIES=true`.
- **Keep the key.** Forge tokens and the backup destination's S3 secret are sealed under a key derived from `SECRET_KEY`, so a copy of the database alone can't read them. Unset, the instance makes a random key and keeps it in `/data/secret_key`, readable by its owner only; keep that volume. If the key changes (say, a restore onto an instance with a different key), Settings marks those credentials and asks for them again; nothing else breaks. The app warns at startup when `SECRET_KEY` is a public placeholder or `/data` is not a mounted volume, and refuses to start with `dev-secret-change-me` when `SECURE_COOKIES=true`.
- **`SECURE_COOKIES=true`** whenever Grimoire is served over HTTPS.
- **Registration** is open only for the first account. Invite people, or open it briefly from Settings.
- **Signing everyone else out**: changing a password ends that user's other sessions; Settings → Account has a button for it too.
+4 -4
View File
@@ -3,9 +3,9 @@
## Running it
```bash
docker compose up --build # app + PostgreSQL at http://localhost:5000
docker compose up --build app # rebuild after a change
docker compose down -v # stop and wipe the database
docker compose -f docker-compose.dev.yml up --build # app + PostgreSQL at http://localhost:5000
docker compose -f docker-compose.dev.yml up --build app # rebuild after a change
docker compose -f docker-compose.dev.yml down -v # stop and wipe the database
```
The image builds the frontend, so a frontend change needs a rebuild too. For quicker frontend work, run Vite against the running app; it proxies `/api` to port 5000:
@@ -32,7 +32,7 @@ Unit tests run without a database. Tests marked `integration` need a real Postgr
The container runs `alembic upgrade head` on every start, after waiting for the database to accept connections.
```bash
docker compose exec app alembic revision -m "short_description"
docker compose -f docker-compose.dev.yml exec app alembic revision -m "short_description"
```
- Number revisions sequentially (`0123_...`). Revision ids must fit in 32 characters, the width of `alembic_version.version_num`.
+18 -4
View File
@@ -1,11 +1,12 @@
import logging
import os
import time
import traceback as tb_module
from pathlib import Path
from quart import Quart, g, jsonify, make_response, request, send_from_directory
from grimoire.config import Config
from grimoire.config import PLACEHOLDER_SECRET_KEYS, SECRET_KEY_FILENAME, Config
from grimoire.routes.admin import admin_bp
from grimoire.routes.api import api
from grimoire.routes.auth import auth_bp
@@ -62,10 +63,23 @@ def create_app() -> Quart:
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
app.config["SESSION_COOKIE_SECURE"] = Config.SECURE_COOKIES
if Config.SECRET_KEY == "dev-secret-change-me":
if Config.SECRET_KEY in PLACEHOLDER_SECRET_KEYS:
logger.warning(
"SECRET_KEY is set to the default value — session cookies are insecure. "
"Set SECRET_KEY or SECRET_KEY_FILE for production use."
"SECRET_KEY is set to a public placeholder, so session cookies are insecure. "
"Remove it to let the instance keep its own key in %s, or set a real one.",
Config.DATA_DIR,
)
if Config.SECRET_KEY_SOURCE == "ephemeral":
logger.warning(
"No SECRET_KEY is set and %s does not exist, so this process is using a "
"temporary key. Sessions and saved credentials will not survive a restart.",
Config.DATA_DIR,
)
elif Config.SECRET_KEY_SOURCE == "data" and not os.path.ismount(Config.DATA_DIR):
logger.warning(
"%s is not a mounted volume. The instance's key (%s) and the model cache "
"are lost when the container is recreated; mount a volume there.",
Config.DATA_DIR, SECRET_KEY_FILENAME,
)
if not Config.TRUST_PROXY_HEADERS:
+76 -3
View File
@@ -1,4 +1,13 @@
import os
import secrets
from pathlib import Path
# The file in the data directory that holds the instance's own SECRET_KEY.
SECRET_KEY_FILENAME = "secret_key"
# Placeholder keys that sample files and older compose files have shipped. A
# key anyone can read in this repository signs nothing safely.
PLACEHOLDER_SECRET_KEYS = frozenset({"dev-secret-change-me", "change-me-in-production"})
def _read_secret(env_var: str, secret_file_var: str, default: str) -> str:
@@ -16,13 +25,76 @@ def _read_secret(env_var: str, secret_file_var: str, default: str) -> str:
return default
def load_or_create_secret_key(data_dir: Path) -> str | None:
"""The instance's own SECRET_KEY, kept in `data_dir` and made on first use.
The first call writes a random key to `<data_dir>/secret_key`, readable by
its owner only; every later call, from this process or another, reads that
same key back. The file is published with a hard link, so two processes
starting together agree on one key and a crash never leaves an empty file.
Answers None when `data_dir` doesn't exist, which is a run outside the
container. An empty key file raises rather than being replaced, because a
new key would orphan everything sealed under the old one.
"""
if not data_dir.is_dir():
return None
path = data_dir / SECRET_KEY_FILENAME
if not path.exists():
tmp = data_dir / f".{SECRET_KEY_FILENAME}.{os.getpid()}"
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
with os.fdopen(fd, "w") as f:
f.write(secrets.token_hex(32) + "\n")
f.flush()
os.fsync(f.fileno())
try:
os.link(tmp, path)
except FileExistsError:
pass
finally:
tmp.unlink(missing_ok=True)
key = path.read_text().strip()
if not key:
raise RuntimeError(
f"{path} is empty. Restore it from a copy, or delete it to start "
"with a new key (saved forge tokens and the backup secret are then "
"entered again)."
)
return key
def _secret_key() -> tuple[str, str]:
"""SECRET_KEY and where it came from: "env", "data" or "ephemeral".
An explicit key (SECRET_KEY or SECRET_KEY_FILE) wins and leaves the data
directory's key file alone, so unsetting it returns to that key. With no
explicit key the instance keeps its own in DATA_DIR. With neither, the
process gets a random key of its own that dies with it.
"""
explicit = _read_secret("SECRET_KEY", "SECRET_KEY_FILE", "")
if explicit:
return explicit, "env"
if os.environ.get("SECRET_KEY_FILE"):
# Falling through to another key would orphan what the named one sealed.
raise RuntimeError(
f"SECRET_KEY_FILE={os.environ['SECRET_KEY_FILE']} is missing or empty."
)
kept = load_or_create_secret_key(Path(os.environ.get("DATA_DIR", "/data")))
if kept:
return kept, "data"
return secrets.token_hex(32), "ephemeral"
class Config:
DATABASE_URL: str = _read_secret(
"DATABASE_URL",
"DATABASE_URL_FILE",
"postgresql+asyncpg://grimoire:grimoire@localhost:5432/grimoire",
)
SECRET_KEY: str = _read_secret("SECRET_KEY", "SECRET_KEY_FILE", "dev-secret-change-me")
# Where the instance keeps its own SECRET_KEY. /data is a volume in every
# shipped compose file, beside the model cache and diagnostics.
DATA_DIR: str = os.environ.get("DATA_DIR", "/data")
SECRET_KEY, SECRET_KEY_SOURCE = _secret_key()
SECURE_COOKIES: bool = os.environ.get("SECURE_COOKIES", "").lower() in ("1", "true", "yes")
LOG_LEVEL: str = os.environ.get("LOG_LEVEL", "INFO")
LOG_RETENTION_DAYS: int = int(os.environ.get("LOG_RETENTION_DAYS", "90"))
@@ -93,8 +165,9 @@ class Config:
errors.append(f"BASE_URL='{cls.BASE_URL}' must start with http:// or https:// when OIDC is enabled")
if cls.SECRET_KEY == "dev-secret-change-me" and cls.SECURE_COOKIES:
errors.append(
"SECRET_KEY is set to the insecure default but SECURE_COOKIES=true indicates "
"a production deployment. Set SECRET_KEY or SECRET_KEY_FILE before starting."
"SECRET_KEY is set to the public placeholder 'dev-secret-change-me' but "
"SECURE_COOKIES=true indicates a production deployment. Remove SECRET_KEY "
"to let the instance keep its own key in DATA_DIR, or set a real one."
)
if errors:
raise ValueError("Configuration errors:\n" + "\n".join(f" - {e}" for e in errors))
+83
View File
@@ -0,0 +1,83 @@
"""The instance keeps its own SECRET_KEY in the data directory.
A deployment that never sets SECRET_KEY must get the same key on every start,
or everything sealed under the first one (forge tokens, the backup secret)
stops opening after the first container recreate.
"""
import stat
from unittest.mock import patch
import pytest
from grimoire import config
from grimoire.config import SECRET_KEY_FILENAME, load_or_create_secret_key
from grimoire.services import secrets
def test_the_first_start_writes_a_key_only_its_owner_can_read(tmp_path):
key = load_or_create_secret_key(tmp_path)
path = tmp_path / SECRET_KEY_FILENAME
assert key and len(key) == 64
assert path.read_text().strip() == key
assert stat.S_IMODE(path.stat().st_mode) == 0o600
assert [p.name for p in tmp_path.iterdir()] == [SECRET_KEY_FILENAME]
def test_every_later_start_reads_the_same_key_and_what_it_sealed_still_opens(tmp_path):
first = load_or_create_secret_key(tmp_path)
with patch.object(secrets.Config, "SECRET_KEY", first):
sealed = secrets.seal("forge-token")
second = load_or_create_secret_key(tmp_path)
assert second == first
with patch.object(secrets.Config, "SECRET_KEY", second):
assert secrets.unseal(sealed) == "forge-token"
def test_no_data_directory_means_no_kept_key(tmp_path):
assert load_or_create_secret_key(tmp_path / "absent") is None
def test_an_empty_key_file_is_refused_rather_than_replaced(tmp_path):
(tmp_path / SECRET_KEY_FILENAME).write_text("\n")
with pytest.raises(RuntimeError, match="is empty"):
load_or_create_secret_key(tmp_path)
assert (tmp_path / SECRET_KEY_FILENAME).read_text() == "\n"
def test_an_explicit_key_wins_and_leaves_the_kept_one_alone(tmp_path, monkeypatch):
monkeypatch.setenv("DATA_DIR", str(tmp_path))
monkeypatch.delenv("SECRET_KEY_FILE", raising=False)
monkeypatch.delenv("SECRET_KEY", raising=False)
kept, source = config._secret_key()
assert source == "data"
monkeypatch.setenv("SECRET_KEY", "an-operator-key")
assert config._secret_key() == ("an-operator-key", "env")
monkeypatch.delenv("SECRET_KEY")
assert config._secret_key() == (kept, "data")
def test_a_secret_key_file_that_is_missing_stops_the_start(tmp_path, monkeypatch):
monkeypatch.setenv("DATA_DIR", str(tmp_path))
monkeypatch.delenv("SECRET_KEY", raising=False)
monkeypatch.setenv("SECRET_KEY_FILE", str(tmp_path / "no-such-secret"))
with pytest.raises(RuntimeError, match="SECRET_KEY_FILE"):
config._secret_key()
assert not (tmp_path / SECRET_KEY_FILENAME).exists()
def test_with_nowhere_to_keep_a_key_the_process_gets_a_temporary_one(tmp_path, monkeypatch):
monkeypatch.setenv("DATA_DIR", str(tmp_path / "absent"))
monkeypatch.delenv("SECRET_KEY", raising=False)
monkeypatch.delenv("SECRET_KEY_FILE", raising=False)
key, source = config._secret_key()
assert source == "ephemeral"
assert key not in config.PLACEHOLDER_SECRET_KEYS and len(key) == 64
+24
View File
@@ -0,0 +1,24 @@
"""The compose file the README shows is the one users deploy from.
The README carries docker-compose.yml inline, so a newcomer can copy it from
the page. Both copies have to say the same thing.
"""
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
def test_the_readme_shows_the_shipped_compose_file_verbatim():
compose = (ROOT / "docker-compose.yml").read_text()
readme = (ROOT / "README.md").read_text()
assert f"```yaml\n{compose}```" in readme
def test_the_shipped_compose_needs_no_key_and_keeps_its_data():
compose = (ROOT / "docker-compose.yml").read_text()
settings = [line for line in compose.splitlines() if not line.lstrip().startswith("#")]
assert not [line for line in settings if "SECRET_KEY" in line]
assert "app_data:/data" in compose
assert "pgdata:/var/lib/postgresql/data" in compose