Merge pull request 'Restore rolls back to a backup; a session can only file issues in another project' (#220) from dev into main
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m44s
CI & Build / integration (push) Successful in 1m58s
CI & Build / Build & push image (push) Successful in 30s

This commit was merged in pull request #220.
This commit is contained in:
2026-11-05 09:29:54 -05:00
52 changed files with 3853 additions and 729 deletions
+6 -7
View File
@@ -28,10 +28,10 @@ services:
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
# Behind an HTTPS reverse proxy:
# Port 5000 is published straight to clients, so no proxy is trusted.
# Behind a reverse proxy, set 1 (one per proxy) and BASE_URL:
TRUSTED_PROXY_HOPS: "0"
# BASE_URL: "https://grimoire.example.com"
# SECURE_COOKIES: "true"
# TRUST_PROXY_HEADERS: "true"
volumes:
- app_data:/data
depends_on:
@@ -75,8 +75,7 @@ Everything is optional. Set these in the environment or in a `.env` file beside
| Variable | When | What it does |
|----------|------|--------------|
| `BASE_URL` | Behind a proxy | The public URL, used in email links and SSO redirects |
| `SECURE_COOKIES` | Over HTTPS | `true` when Grimoire is served over HTTPS |
| `TRUST_PROXY_HEADERS` | Behind a proxy | `true` to read the client address from the proxy's headers |
| `TRUSTED_PROXY_HOPS` | Behind a proxy | How many reverse proxies stand in front of Grimoire (default `1`; the compose file above sets `0` because it publishes the port directly). The client address and HTTPS are read from the proxy's headers by it |
| `POSTGRES_PASSWORD` | Before the first start | The database password (defaults to `grimoire`; the database is reachable only inside the compose network). Postgres takes it once, when the volume is created |
| `SECRET_KEY` | Rarely | A key you manage yourself, instead of the one Grimoire keeps. It wins over the kept key; remove it to go back |
@@ -97,7 +96,7 @@ docker compose up -d
Database migrations run when the app starts, and the key stays in its volume. Claude Code updates the plugin from its marketplace; run `/reload-plugins` after an update.
**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](docs/backups.md).
**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and Settings → Data → Restore from Backup rolls the instance back to one, or brings a new instance back from it. See [Backups](docs/backups.md).
## Features
@@ -132,7 +131,7 @@ The full list is in [Features](docs/features.md).
| What you see | Why | Fix |
|--------------|-----|-----|
| `SECRET_KEY is set to the public placeholder` at startup | `SECURE_COOKIES=true` while `SECRET_KEY` is `dev-secret-change-me` | Remove `SECRET_KEY`, or set a real one |
| `SECRET_KEY is set to the public placeholder` at startup | `SECRET_KEY` is `dev-secret-change-me` or `change-me-in-production` | Remove `SECRET_KEY`, or set a real one |
| Settings asks for forge tokens or the backup secret again | The instance's key changed: the `app_data` volume was replaced, or `SECRET_KEY` was set or removed | Enter them again, or restore the previous key |
| `Can't locate revision identified by '…'` at startup | An older image is running against a database a newer one has already migrated | Run the newer image; migrations only move forward |
| The plugin's tools are missing in Claude Code | The plugin isn't installed, or Claude Code hasn't reloaded it | `/plugin` to install or update it, then `/reload-plugins` |
@@ -0,0 +1,39 @@
"""instance identity — the id lineage a backup carries
Revision ID: 0127
Revises: 0126
Create Date: 2026-11-05
One row naming this database's id lineage. Every full backup carries it, a
restore into a fresh instance adopts it (the restore keeps the original ids),
and rolling a live instance back requires the backup's to match, so a backup's
ids never land on unrelated rows. Made here with a random value, so every
instance has one before its first account exists.
"""
import sqlalchemy as sa
from alembic import op
revision = "0127"
down_revision = "0126"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"instance_identity",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("instance_id", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.CheckConstraint("id = 1", name="ck_instance_identity_single_row"),
)
op.execute(
"INSERT INTO instance_identity (id, instance_id) VALUES (1, gen_random_uuid()::text)"
)
def downgrade() -> None:
op.drop_table("instance_identity")
+3 -2
View File
@@ -8,8 +8,9 @@ services:
# a Docker secret and sets SECRET_KEY_FILE to its path).
SECRET_KEY: "${SECRET_KEY}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
TRUST_PROXY_HEADERS: "true"
SECURE_COOKIES: "true"
# Proxies in front of the app (Traefik = 1). The client address and
# HTTPS are read from that far in from the right of X-Forwarded-*.
TRUSTED_PROXY_HOPS: "1"
volumes:
- app_data:/data
networks:
+3 -3
View File
@@ -13,10 +13,10 @@ services:
environment:
DATABASE_URL: "postgresql+asyncpg://${POSTGRES_USER:-grimoire}:${POSTGRES_PASSWORD:-grimoire}@db:5432/${POSTGRES_DB:-grimoire}"
LOG_LEVEL: "${LOG_LEVEL:-INFO}"
# Behind an HTTPS reverse proxy:
# Port 5000 is published straight to clients, so no proxy is trusted.
# Behind a reverse proxy, set 1 (one per proxy) and BASE_URL:
TRUSTED_PROXY_HOPS: "0"
# BASE_URL: "https://grimoire.example.com"
# SECURE_COOKIES: "true"
# TRUST_PROXY_HEADERS: "true"
volumes:
- app_data:/data
depends_on:
+4 -1
View File
@@ -251,6 +251,7 @@ These serve the plugin's hooks.
| POST | `/api/plugin/rule-opened` | Records that a session opened a rule it was shown |
| GET | `/api/plugin/shape-check` | The end-of-turn question about what this turn wrote |
| GET | `/api/plugin/claim-session` · `/release-session` | Bind a session to its task claims / release them at session end |
| GET | `/api/plugin/write-scope` | Which projects a pending act would change (a directory, or an MCP call's `tool` and `input`), and whether a session in its own project may do it. Filing an issue elsewhere is allowed; every other cross-project write is answered `allowed: false` with the reason the operator is asked |
| GET | `/api/plugin/processes` | Stored Processes for skill-stub sync |
| GET | `/api/plugin/prior-art` | Write-path hint for the plugin hooks (params: `path`, `code`, `repo`, `shapes`, `exclude_ids`, `exclude_sync_ids`, `exclude_derive`); returns `context`, `note_ids`, `sync_note_ids`, `stamped`, `divergence`, `derive`, `derive_keys` |
| GET / POST | `/api/projects/<id>/coverage`, `…/coverage/refresh` | Shape-ledger accounting (`pattern_coverage` line, counts, `derive_groups` — css groups carry `consumers`, `derive_new`, `unused_css`, `divergence`, `recheck`) |
@@ -272,7 +273,9 @@ These serve the plugin's hooks.
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/backup` | Export backup (format v4; `?scope=user` for own data) |
| POST | `/api/admin/restore` | Restore from a backup |
| GET | `/api/admin/restore/sources` | The backups at the backup destination, newest first |
| POST | `/api/admin/restore/preview` | What restoring a backup (`?source=<name>` or an uploaded file) would change; 409 with the reason when it can't apply here |
| POST | `/api/admin/restore?confirm=rollback` | Put the whole instance back to that backup, after saving the current state |
| GET / DELETE | `/api/admin/users` · `/api/admin/users/:id` | List / delete users |
| GET / PUT | `/api/admin/registration` | Get / toggle registration |
| GET / PUT | `/api/admin/smtp` · POST `/api/admin/smtp/test` | SMTP config + test email |
+3 -3
View File
@@ -69,10 +69,10 @@ Every database wait on startup has a deadline. Migrations wait for the database
## Authentication
- **Sessions**: `HttpOnly`, `SameSite=Lax` cookies, `Secure` when `SECURE_COOKIES=true`. Changing a password bumps the user's session version and signs out their other sessions.
- **Sessions**: `HttpOnly`, `SameSite=Lax` cookies, `Secure` on a request the trusted proxy reports as HTTPS, which also gets HSTS (`grimoire/proxy.py`). Changing a password bumps the user's session version and signs out their other sessions.
- **API keys**: `Authorization: Bearer fmcp_...`, stored hashed. A `read` key may only call read endpoints and read MCP tools.
- **OIDC** single sign-on with PKCE, optionally replacing local logins ([SSO / OAuth](sso-oauth.md)).
- **Rate limits** on register, login and forgot-password, per client IP (`TRUST_PROXY_HEADERS` reads the IP from the proxy's headers).
- **Rate limits** on register, login and forgot-password, per client IP, read from `X-Forwarded-For` by `TRUSTED_PROXY_HOPS`.
Credentials Grimoire stores for outside services (forge tokens, the backup destination's S3 secret) are sealed under a key derived from `SECRET_KEY`, which the instance makes on first start and keeps in `/data/secret_key` unless one is set.
@@ -87,7 +87,7 @@ Credentials Grimoire stores for outside services (forge tokens, the backup desti
│ ├── routes/ REST blueprints
│ ├── mcp/ MCP server and its tool modules
│ ├── services/ shared logic
│ └── restore.py `python -m grimoire.restore` for a fresh instance
│ └── restore.py `python -m grimoire.restore`: the restore, from the command line
├── frontend/src/ views, components, stores, composables, api client
├── plugin/ Claude Code hooks, skills and commands
├── scripts/ CI checks and the plugin version minter
+52 -21
View File
@@ -88,9 +88,12 @@ You need four things:
- **Access key.**
- **Secret key.**
Grimoire uploads and lists; it never deletes. A key that may **only add
objects** is therefore enough, and it is the safer key to hand out: a stolen
one cannot wipe the history.
Grimoire uploads and lists, and reads a backup back when you restore from the
bucket; it never deletes. A key that may **add and read objects but not
delete them** is therefore enough, and it is the safer key to hand out: a
stolen one cannot wipe the history. A key that may only add objects still
takes backups; restore then works by uploading a file or from the command
line.
The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
never shown again, and never included in a backup. If an instance's
@@ -122,8 +125,10 @@ keeps service backups as "backup targets". Either destination works with it.
1. On Archives' **Backups** page, make a target for Grimoire, named `grimoire`
for example.
2. Choose how Grimoire's files arrive:
- **S3**: turn on the S3 way in for the target and make an **append-only**
S3 key. In Grimoire, choose **An S3 bucket**:
- **S3**: turn on the S3 way in for the target and make an S3 key that
may add and read objects but not delete them (an **append-only** key
takes backups too, but then Grimoire can't restore from the bucket). In
Grimoire, choose **An S3 bucket**:
- address: Archives' backup address (`http://<archives>:8081`);
- bucket: the target's name;
- access key and secret key: the key you just made.
@@ -139,26 +144,52 @@ keeps service backups as "backup targets". Either destination works with it.
## Restoring
### Into a fresh instance
A restore puts the **whole instance** back to the moment a backup was taken,
for every user:
Start Grimoire on an empty database, copy the backup file into the container,
and run:
- records edited since go back to how they were;
- records deleted since come back;
- anything made since is removed.
It matches records by their ids, so a live instance only restores a backup **of
its own**. Each instance has an id, and every backup records the one it came
from. A backup from another instance, or one made before backups recorded it,
only goes into a new, empty instance, which takes it as it is.
Before changing anything, the restore saves the current state as one more
backup. It goes to the backup destination when scheduled backups are on, and
otherwise to `/data/pre-restore`. The result says which file it is, so a
restore can itself be undone.
What a backup doesn't carry stays as it is wherever its owner survives the
restore: API keys, forge connections, sharing, logs, and the destination's S3
secret. The trash isn't in backups, so anything in the trash when the backup
was taken is gone afterwards. Everyone is signed out, and search is rebuilt.
### From the web
**Settings → Data → Restore from Backup** lists the backups at the backup
destination. You can pick one of those, or upload a `.tar` or the JSON
**Full Backup** download (uploads are held to the server's request-size
limit). The restore then shows what would change: when the backup was taken,
and how many accounts, projects, plans, notes and rules there are now and will
be afterwards. Type `roll back` to run it. Then sign in again.
To read backups back from an S3 bucket, the key needs read access as well as
write. Grimoire never deletes from the bucket.
### From the command line
For a file with no size limit, or a new instance with no backup destination
yet, copy the file into the container and run:
```sh
docker exec -it grimoire python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
```
- The command refuses a database that already has an account. A restore
re-creates everything with new ids, so on a live instance it would add a
second copy instead of replacing anything.
- Nothing limits the file's size.
- It also reads the JSON from the admin **Full Backup** download.
It prints what would change and stops. Add `--yes` to restore. It reads the
JSON download too. Afterwards, **restart Grimoire** so the search indexes are
rebuilt.
Afterwards, **restart Grimoire** so the search indexes are rebuilt. Then issue
new API keys, add forge connections again, and set up any sharing.
### From the web
**Settings → Data → Restore from Backup** takes the `.tar` as well as the
JSON download. The upload is held to the server's request-size limit. For a
bigger file, use the command above.
After restoring into a new instance, issue new API keys, add forge connections
again, and set up any sharing: none of them travel in a backup.
+10 -8
View File
@@ -14,8 +14,7 @@ Any variable marked *secret* can also be read from a file: set `<NAME>_FILE` to
| `SECRET_KEY` *(secret)* | kept in `DATA_DIR` | Signs sessions and seals stored credentials. Unset, the instance makes its own on first start and keeps it in `DATA_DIR/secret_key`. Set it only to manage the key yourself; it wins over the kept one (see below) |
| `DATA_DIR` | `/data` | Where the instance keeps its own `SECRET_KEY`. Mount a volume here |
| `BASE_URL` | `http://localhost:5000` | Public URL, used in email links and OIDC redirects |
| `SECURE_COOKIES` | `false` | `true` when Grimoire is served over HTTPS |
| `TRUST_PROXY_HEADERS` | `false` | `true` behind a reverse proxy, to read the client IP from `X-Forwarded-For` / `X-Real-IP` |
| `TRUSTED_PROXY_HOPS` | `1` | How many reverse proxies stand in front of Grimoire. The client address and the scheme are read that many entries in from the right of `X-Forwarded-For` and `X-Forwarded-Proto`; `0` ignores both headers. See [Behind a reverse proxy](#behind-a-reverse-proxy) |
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING` or `ERROR` |
| `LOG_RETENTION_DAYS` | `90` | Days of app logs to keep |
| `FASTEMBED_CACHE_DIR` | `/data/fastembed-cache` | Where the embedding model is stored after its first download |
@@ -77,7 +76,7 @@ docker compose -f docker-compose.dev.yml up --build
### Docker Swarm
[`docker-compose.prod.yml`](../docker-compose.prod.yml) is a stack file: the app and database on a private overlay network, with `SECURE_COOKIES` and `TRUST_PROXY_HEADERS` on. A Swarm can move the app to another node, away from a node-local `/data` volume, so the stack sets its own key: provide `DB_PASSWORD` and `SECRET_KEY` in the environment (or mount a Docker secret and set `SECRET_KEY_FILE`):
[`docker-compose.prod.yml`](../docker-compose.prod.yml) is a stack file: the app and database on a private overlay network, trusting one proxy (`TRUSTED_PROXY_HOPS: "1"`). A Swarm can move the app to another node, away from a node-local `/data` volume, so the stack sets its own key: provide `DB_PASSWORD` and `SECRET_KEY` in the environment (or mount a Docker secret and set `SECRET_KEY_FILE`):
```bash
DB_PASSWORD=... SECRET_KEY=... docker stack deploy -c docker-compose.prod.yml grimoire
@@ -96,13 +95,12 @@ Back up the database with **Settings → Data → Scheduled backups** ([Backups]
## Behind a reverse proxy
Grimoire doesn't terminate TLS. Put it behind Nginx, Traefik or Caddy, forward to port 5000, and keep port 5000 off the internet. Set `SECURE_COOKIES=true`, `TRUST_PROXY_HEADERS=true` and `BASE_URL`.
Grimoire doesn't terminate TLS. Put it behind Nginx, Traefik or Caddy, forward to port 5000, and keep port 5000 off the internet. Set `BASE_URL`, and `TRUSTED_PROXY_HOPS` to the number of proxies a request passes through (one proxy is `1`, the default; Cloudflare in front of Traefik is `2`).
```nginx
location / {
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
@@ -112,12 +110,16 @@ location / {
}
```
With `TRUST_PROXY_HEADERS=true`, make sure the proxy replaces any `X-Forwarded-For` the client sends.
Each proxy appends its entry to `X-Forwarded-For` and `X-Forwarded-Proto`, so Grimoire reads the entry `TRUSTED_PROXY_HOPS` in from the right: the one your own proxy wrote. Whatever a client puts to the left of it is ignored. A header with fewer entries than that (a LAN road with one proxy beside a Cloudflare road with two) is read at its leftmost entry, the one the first proxy wrote.
When that entry says `https`, the session cookie is marked `Secure` and the response carries `Strict-Transport-Security: max-age=31536000`. Over plain HTTP neither is sent, and Grimoire never redirects to HTTPS, so a LAN install keeps working.
A hop count higher than the real number of proxies lets a client choose its own address, so count them. With `0`, the headers are ignored and the socket peer is the client: use it when clients reach port 5000 directly.
## Security checklist
- **Keep the key.** Forge tokens and the backup destination's S3 secret are sealed under a key derived from `SECRET_KEY`, so a copy of the database alone can't read them. Unset, the instance makes a random key and keeps it in `/data/secret_key`, readable by its owner only; keep that volume. If the key changes (say, a restore onto an instance with a different key), Settings marks those credentials and asks for them again; nothing else breaks. The app warns at startup when `SECRET_KEY` is a public placeholder or `/data` is not a mounted volume, and refuses to start with `dev-secret-change-me` when `SECURE_COOKIES=true`.
- **`SECURE_COOKIES=true`** whenever Grimoire is served over HTTPS.
- **Keep the key.** Forge tokens and the backup destination's S3 secret are sealed under a key derived from `SECRET_KEY`, so a copy of the database alone can't read them. Unset, the instance makes a random key and keeps it in `/data/secret_key`, readable by its owner only; keep that volume. If the key changes (say, a restore onto an instance with a different key), Settings marks those credentials and asks for them again; nothing else breaks. The app warns at startup when `SECRET_KEY` is a public placeholder or `/data` is not a mounted volume, and refuses to start with a placeholder key (`dev-secret-change-me`, `change-me-in-production`).
- **`TRUSTED_PROXY_HOPS`** matches the number of proxies in front of Grimoire, so rate limits and the audit log see the real client and the cookie is `Secure` over HTTPS.
- **Registration** is open only for the first account. Invite people, or open it briefly from Settings.
- **Signing everyone else out**: changing a password ends that user's other sessions; Settings → Account has a button for it too.
- **API keys** are per user, shown once, and can be read-only. Revoke any you no longer use under Settings → MCP Access.
+3 -2
View File
@@ -66,6 +66,7 @@ Saved procedures, such as a drift audit or a DRY pass. The plugin syncs each one
The whole store is reachable through the MCP endpoint at `/mcp`, authenticated with an API key. The Claude Code plugin in this repo adds:
- hooks for session start, each prompt, before and after tool calls, the end of a turn, compaction and session end, all failing open if Grimoire is unreachable;
- a project fence: a session belongs to the project its starting directory is bound to. Toward any other project it can read anything and file an issue. Editing that project's files, committing or pushing in its repo, or changing its records first asks you, in every permission mode;
- skills for the working practices: using Grimoire, writing plans, systematic debugging, verification, brainstorming, reporting back, reusing code, shape accounting and family canon;
- your stored processes as skills.
@@ -85,8 +86,8 @@ Share a project, note or task with users or groups, as viewer, editor or admin.
## Export and backup
- **Personal export**: your notes and tasks as a Markdown ZIP (with YAML frontmatter) or JSON.
- **Admin backup and restore**, with ids remapped so a backup restores onto another instance.
- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](backups.md).
- **Admin backup and restore**: a restore rolls the whole instance back to one of its own backups, after saving the current state, or brings a new instance back from any backup.
- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. Restoring is in Settings, or `python -m grimoire.restore`. See [Backups](backups.md).
## Sign-in
+70 -1
View File
@@ -1,4 +1,4 @@
import { apiGet, apiPost, apiPut } from "@/api/client";
import { apiGet, apiPost, apiPostRaw, apiPut } from "@/api/client";
/**
* Scheduled backups — `/api/admin/backups`. One dated,
@@ -71,3 +71,72 @@ export function checkBackupDestination(
export function runBackupNow(): Promise<{ status: string }> {
return apiPost<{ status: string }>("/api/admin/backups/run", {});
}
// ── Restore: put the whole instance back to a full backup ─────────────────
export interface StoredBackup {
name: string;
size_bytes: number;
}
export interface RestoreCount {
kind: string;
now: number;
after: number;
}
export interface RestorePreview {
/** "fresh": an empty instance takes the backup. "rollback": this instance
* goes back to its own earlier backup. */
mode: "fresh" | "rollback";
exported_at: string | null;
counts: RestoreCount[];
}
export interface RestoreResult extends RestorePreview {
status: "ok";
restored: Record<string, number>;
safety_backup: { file: string; size_bytes: number; where: string; note?: string };
}
/** A backup at the destination, by name, or a file picked from this computer. */
export type RestoreFrom = { source: string } | { file: File };
// Reading, checking and writing the whole store is slow by design; bounded,
// but in minutes.
const RESTORE_TIMEOUT_MS = 10 * 60 * 1000;
export function listRestoreSources(): Promise<{ backups: StoredBackup[]; error?: string }> {
return apiGet("/api/admin/restore/sources", { timeoutMs: 60_000 });
}
async function restoreCall<T>(path: string, from: RestoreFrom, query: Record<string, string> = {}): Promise<T> {
const params = new URLSearchParams(query);
if ("source" in from) {
params.set("source", from.source);
return apiPostRaw<T>(`${path}?${params}`, null, null, { timeoutMs: RESTORE_TIMEOUT_MS });
}
const qs = params.toString() ? `?${params}` : "";
const isTar = from.file.name.toLowerCase().endsWith(".tar");
// The JSON download is parsed first, so a damaged file fails here, before
// the upload.
const body = isTar ? from.file : JSON.stringify(JSON.parse(await from.file.text()));
return apiPostRaw<T>(`${path}${qs}`, body, isTar ? "application/x-tar" : "application/json",
{ timeoutMs: RESTORE_TIMEOUT_MS });
}
export function previewRestore(from: RestoreFrom): Promise<RestorePreview> {
return restoreCall<RestorePreview>("/api/admin/restore/preview", from);
}
export function restoreBackup(from: RestoreFrom): Promise<RestoreResult> {
return restoreCall<RestoreResult>("/api/admin/restore", from, { confirm: "rollback" });
}
/** When a backup named `grimoire-20261015T020000Z.tar` was taken. */
export function backupTakenAt(name: string): Date | null {
const m = /(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z/.exec(name);
if (!m) return null;
return new Date(Date.UTC(+m[1], +m[2] - 1, +m[3], +m[4], +m[5], +m[6]));
}
+9
View File
@@ -156,6 +156,15 @@ export function apiPost<T>(path: string, body: unknown, opts?: RequestOpts): Pro
return request<T>(path, { method: "POST", headers: JSON_HEADERS, body: JSON.stringify(body) }, opts);
}
/** A POST whose body is sent as it is (a file, or text already serialised),
* under the same deadline and error handling as every other call. */
export function apiPostRaw<T>(
path: string, body: BodyInit | null, contentType: string | null, opts?: RequestOpts,
): Promise<T> {
const headers = contentType ? { "Content-Type": contentType } : undefined;
return request<T>(path, { method: "POST", headers, body }, opts);
}
export function apiPut<T>(path: string, body: unknown, opts?: RequestOpts): Promise<T> {
return request<T>(path, { method: "PUT", headers: JSON_HEADERS, body: JSON.stringify(body) }, opts);
}
@@ -0,0 +1,278 @@
<script setup lang="ts">
/**
* Restore: put the whole instance back to a full backup.
*
* Choose a backup (one at the backup destination, or a file from this
* computer), see what it would change, type the confirmation, run it. On a
* live instance this is a rollback to the instance's own earlier backup; on a
* new, empty one it takes the backup in. Either way the server backs up the
* current state first and signs everyone out afterwards.
*/
import { computed, onMounted, ref } from "vue";
import { apiErrorMessage } from "@/api/client";
import {
backupTakenAt, listRestoreSources, previewRestore, restoreBackup,
type RestoreFrom, type RestorePreview, type RestoreResult, type StoredBackup,
} from "@/api/backups";
import { formatBytes } from "@/utils/bytes";
const emit = defineEmits<{ close: [] }>();
const CONFIRM_WORDS = "roll back";
type Step = "choose" | "checking" | "preview" | "running" | "done";
const step = ref<Step>("choose");
const sources = ref<StoredBackup[]>([]);
const sourcesError = ref("");
const sourcesLoading = ref(true);
const chosen = ref<RestoreFrom | null>(null);
const chosenLabel = ref("");
const preview = ref<RestorePreview | null>(null);
const result = ref<RestoreResult | null>(null);
const error = ref("");
const typed = ref("");
const fileInput = ref<HTMLInputElement | null>(null);
const confirmed = computed(() => typed.value.trim().toLowerCase() === CONFIRM_WORDS);
function when(value: Date | string | null | undefined): string {
if (!value) return "an unknown date";
const date = typeof value === "string" ? new Date(value) : value;
return Number.isNaN(date.getTime()) ? "an unknown date" : date.toLocaleString();
}
onMounted(async () => {
try {
const answer = await listRestoreSources();
sources.value = answer.backups;
sourcesError.value = answer.error ?? "";
} catch (e) {
sourcesError.value = apiErrorMessage(e, "Could not list the backups at the destination.");
} finally {
sourcesLoading.value = false;
}
});
async function check(from: RestoreFrom, label: string) {
chosen.value = from;
chosenLabel.value = label;
error.value = "";
typed.value = "";
step.value = "checking";
try {
preview.value = await previewRestore(from);
step.value = "preview";
} catch (e) {
error.value = apiErrorMessage(e, "That backup could not be read.");
step.value = "choose";
}
}
function pickFile(event: Event) {
const file = (event.target as HTMLInputElement).files?.[0];
if (fileInput.value) fileInput.value.value = "";
if (file) check({ file }, file.name);
}
async function run() {
if (!chosen.value || !confirmed.value) return;
error.value = "";
step.value = "running";
try {
result.value = await restoreBackup(chosen.value);
step.value = "done";
} catch (e) {
error.value = apiErrorMessage(e, "The restore failed. Nothing was changed.");
step.value = "preview";
}
}
function finish() {
// Every session ended with the restore, this one included.
window.location.assign("/login");
}
</script>
<template>
<teleport to="body">
<div class="modal-overlay" @click="step === 'running' || step === 'done' ? null : emit('close')">
<div class="modal-card restore-card" @click.stop>
<template v-if="step === 'choose' || step === 'checking'">
<h3 class="modal-title">Restore from a backup</h3>
<p class="modal-message">
Puts the whole instance back to the moment the backup was taken, for every user.
You'll see what changes before anything happens.
</p>
<p v-if="error" class="restore-error" role="alert">{{ error }}</p>
<h4 class="restore-subhead">At the backup destination</h4>
<p v-if="sourcesLoading" class="restore-muted">Looking for backups…</p>
<p v-else-if="sourcesError" class="restore-muted">{{ sourcesError }}</p>
<p v-else-if="!sources.length" class="restore-muted">No backups there yet.</p>
<ul v-else class="restore-sources">
<li v-for="b in sources" :key="b.name">
<button
class="restore-source"
:disabled="step === 'checking'"
@click="check({ source: b.name }, when(backupTakenAt(b.name)))"
>
<span>{{ when(backupTakenAt(b.name)) }}</span>
<span class="restore-muted">{{ formatBytes(b.size_bytes) }}</span>
</button>
</li>
</ul>
<h4 class="restore-subhead">From this computer</h4>
<p class="restore-muted">
A backup file (.tar) or the Full Backup download (.json). Uploads are size-limited;
a large file restores from the destination above, or with
<code>python -m grimoire.restore</code>.
</p>
<div class="modal-actions restore-actions-left">
<button class="modal-btn" :disabled="step === 'checking'" @click="fileInput?.click()">
Choose a file…
</button>
<input ref="fileInput" type="file" accept=".json,.tar" class="hidden-file-input" @change="pickFile" />
</div>
<p v-if="step === 'checking'" class="restore-muted">Reading {{ chosenLabel }}…</p>
<div class="modal-actions">
<button class="modal-btn" @click="emit('close')">Cancel</button>
</div>
</template>
<template v-else-if="(step === 'preview' || step === 'running') && preview">
<h3 class="modal-title">
{{ preview.mode === "rollback" ? "Roll back to this backup?" : "Restore this backup?" }}
</h3>
<p class="modal-message">
<template v-if="preview.mode === 'rollback'">
Everything goes back to {{ when(preview.exported_at) }}, for every user. Anything made
since is removed, edits since are undone, and records deleted since come back.
</template>
<template v-else>
This instance has no records yet, so it takes the backup from {{ when(preview.exported_at) }} as it is.
</template>
The current state is backed up first, and everyone is signed out afterwards.
</p>
<table class="restore-counts">
<thead><tr><th></th><th>Now</th><th>After</th></tr></thead>
<tbody>
<tr v-for="c in preview.counts" :key="c.kind">
<td>{{ c.kind }}</td><td>{{ c.now }}</td><td>{{ c.after }}</td>
</tr>
</tbody>
</table>
<p v-if="error" class="restore-error" role="alert">{{ error }}</p>
<label class="restore-confirm">
Type <strong>{{ CONFIRM_WORDS }}</strong> to continue
<input v-model="typed" class="fs-input input" :disabled="step === 'running'" autocomplete="off" />
</label>
<div class="modal-actions">
<button class="modal-btn" :disabled="step === 'running'" @click="step = 'choose'">Back</button>
<button
class="modal-btn modal-btn-danger"
:disabled="!confirmed || step === 'running'"
@click="run"
>{{ step === "running" ? "Restoring…" : preview.mode === "rollback" ? "Roll back" : "Restore" }}</button>
</div>
</template>
<template v-else-if="step === 'done' && result">
<h3 class="modal-title">Restored</h3>
<p class="modal-message">
The instance is back to {{ when(result.exported_at) }}.
The state it replaced was saved as <code>{{ result.safety_backup.file }}</code>
in {{ result.safety_backup.where }}.
<template v-if="result.safety_backup.note"> {{ result.safety_backup.note }}</template>
Search is being rebuilt in the background. Everyone has been signed out, so sign in again.
</p>
<div class="modal-actions">
<button class="modal-btn modal-btn-primary" @click="finish">Sign in again</button>
</div>
</template>
</div>
</div>
</teleport>
</template>
<style scoped>
.restore-card {
max-width: 34rem;
max-height: 85vh;
overflow-y: auto;
}
.restore-subhead {
margin: 1rem 0 0.4rem;
font-size: 0.9rem;
}
.restore-muted {
font-size: 0.85rem;
color: var(--fs-text-secondary);
margin: 0 0 0.5rem;
}
.restore-error {
font-size: 0.85rem;
color: var(--fs-action-destructive);
margin: 0 0 0.75rem;
}
.restore-sources {
list-style: none;
margin: 0 0 0.5rem;
padding: 0;
max-height: 14rem;
overflow-y: auto;
border: 1px solid var(--fs-border-color);
border-radius: var(--fs-radius-sm);
}
.restore-source {
display: flex;
justify-content: space-between;
width: 100%;
padding: 0.45rem 0.7rem;
border: none;
border-bottom: 1px solid var(--fs-border-color);
background: none;
color: var(--fs-text-primary);
font: inherit;
font-size: 0.875rem;
text-align: left;
cursor: pointer;
}
.restore-sources li:last-child .restore-source {
border-bottom: none;
}
.restore-source:hover:not(:disabled) {
background: var(--fs-surface-hover);
}
.restore-actions-left {
justify-content: flex-start;
margin-bottom: 1rem;
}
.restore-counts {
width: 100%;
border-collapse: collapse;
font-size: 0.875rem;
margin: 0 0 1rem;
}
.restore-counts th,
.restore-counts td {
text-align: right;
padding: 0.25rem 0.5rem;
border-bottom: 1px solid var(--fs-border-color);
}
.restore-counts th:first-child,
.restore-counts td:first-child {
text-align: left;
}
.hidden-file-input {
display: none;
}
.restore-confirm {
display: flex;
flex-direction: column;
gap: 0.35rem;
font-size: 0.875rem;
margin: 0 0 1rem;
}
</style>
@@ -0,0 +1,96 @@
<script setup lang="ts">
/**
* The end of a server-paged list: asks for the next page as the reader nears
* it, in place of a "Load more" button. The parent keeps the paging — what the
* next page is keyed on, whether one is in flight, whether the last one
* failed — and this owns the observing.
*
* <ScrollSentinel :cursor="more ? String(rows.length) : ''"
* :loading="loadingMore" :failed="moreFailed" @more="loadMore" />
*
* An empty cursor means the list is complete, and nothing renders.
*/
import { onBeforeUnmount, ref, watch } from "vue";
const props = defineProps<{ cursor: string; loading: boolean; failed?: boolean }>();
const emit = defineEmits<{ more: [] }>();
const sentinel = ref<HTMLElement | null>(null);
const visible = ref(false);
// One request per page. The observer reports every crossing, and a short last
// page leaves the sentinel on screen for good, so the cursor last asked for is
// remembered: a page is asked for once, and a failed one is not retried on
// every scroll, because a failure leaves the cursor where it was. Only
// Try again asks for it again.
let asked = "";
function pump() {
if (!visible.value || props.loading || props.failed || !props.cursor) return;
if (props.cursor === asked) return;
asked = props.cursor;
emit("more");
}
function retry() {
asked = props.cursor;
emit("more");
}
let observer: IntersectionObserver | undefined;
function observe(el: HTMLElement | null) {
observer?.disconnect();
observer = undefined;
if (!el) {
// Visibility and the cursor asked for go with the element. A list that
// starts over must not inherit "the end is on screen" from the one it
// replaced, and an offset cursor repeats from list to list, so "50" asked
// of the old list must not stop the new one asking for its own "50".
visible.value = false;
asked = "";
return;
}
observer = new IntersectionObserver(
(entries) => {
visible.value = entries.some((e) => e.isIntersecting);
pump();
},
// Extends the watched area a screen below the viewport, so the next page is
// asked for before the reader reaches the end rather than when they do.
{ rootMargin: "0px 0px 100% 0px" },
);
observer.observe(el);
}
// flush: "post": the template ref is only set once the DOM is patched.
watch(sentinel, observe, { flush: "post" });
// A page that lands without pushing the end off screen gives the observer
// nothing new to report, so carry on from the props, or the list stops early.
watch([() => props.cursor, () => props.loading, () => props.failed], pump);
onBeforeUnmount(() => observer?.disconnect());
</script>
<template>
<div v-if="cursor" class="sentinel">
<!-- The one case with a button: an observer re-firing on a failure would be
a request loop. -->
<p v-if="failed" class="sentinel-note sentinel-failed" role="alert">
Couldn’t load more.
<button type="button" class="btn-secondary btn-sm" @click="retry">Try again</button>
</p>
<template v-else>
<div ref="sentinel" class="sentinel-mark" aria-hidden="true"></div>
<p v-if="loading" class="sentinel-note" role="status">Loading more…</p>
</template>
</div>
</template>
<style scoped>
.sentinel { grid-column: 1 / -1; padding: 0.75rem 0; text-align: center; }
.sentinel-mark { height: 1px; }
.sentinel-note { margin: 0; font-size: 0.8rem; color: var(--fs-text-tertiary); }
.sentinel-failed { display: flex; align-items: center; justify-content: center; gap: 0.75rem; }
</style>
+21 -7
View File
@@ -19,6 +19,7 @@ import {
type FamilyIdea, type IdeaSnapshot, type IdeaStatus,
} from "@/api/family";
import FamilyAdoptionMatrix from "@/components/FamilyAdoptionMatrix.vue";
import ScrollSentinel from "@/components/ScrollSentinel.vue";
import { useToastStore } from "@/stores/toast";
import { fmtStamp } from "@/utils/dateFormat";
import { recordHref } from "@/utils/recordHref";
@@ -36,6 +37,8 @@ const loading = ref(true);
const failed = ref(false);
const moreDecisions = ref(false);
const loadingMore = ref(false);
const moreFailed = ref(false);
const decisionOffset = ref(0);
// One pending reversal at a time: which row, which act, and its reason.
const pending = ref<{ kind: "undo" | "retire"; id: number } | null>(null);
@@ -63,7 +66,9 @@ async function load() {
criteria.value = ideaData.criteria;
conflictOrder.value = ideaData.conflict_order ?? [];
decisions.value = log;
decisionOffset.value = log.length;
moreDecisions.value = log.length === PAGE;
moreFailed.value = false;
} catch {
// Said out loud: a log that failed to load must not read as "nothing
// has been decided".
@@ -73,14 +78,20 @@ async function load() {
}
}
// Older decisions load as the log is scrolled. The log is offset-paged, so a
// decision recorded since the first page shifts the rest down by one; a row
// already shown is skipped, and the offset still counts what the server sent.
async function loadMore() {
loadingMore.value = true;
moreFailed.value = false;
try {
const next = await listFamilyDecisions(PAGE, decisions.value.length);
decisions.value.push(...next);
const next = await listFamilyDecisions(PAGE, decisionOffset.value);
const shown = new Set(decisions.value.map((d) => d.id));
decisions.value.push(...next.filter((d) => !shown.has(d.id)));
decisionOffset.value += next.length;
moreDecisions.value = next.length === PAGE;
} catch (e: unknown) {
toast.show(apiErrorMessage(e, "Could not load more decisions"), "error");
} catch {
moreFailed.value = true;
} finally {
loadingMore.value = false;
}
@@ -343,9 +354,12 @@ onMounted(load);
</form>
</li>
</ul>
<button v-if="moreDecisions" class="btn-secondary btn-sm" :disabled="loadingMore" @click="loadMore">
{{ loadingMore ? "Loading…" : "Older decisions" }}
</button>
<ScrollSentinel
:cursor="moreDecisions ? String(decisionOffset) : ''"
:loading="loadingMore"
:failed="moreFailed"
@more="loadMore"
/>
</section>
</template>
</main>
+50 -67
View File
@@ -1,5 +1,5 @@
<script setup lang="ts">
import { ref, computed, watch, onMounted, onUnmounted, nextTick } from "vue";
import { ref, computed, watch, onMounted, onUnmounted } from "vue";
import { useRouter } from "vue-router";
import { apiGet } from "@/api/client";
import type { TaskKind, TaskStatus, TaskPriority } from "@/types/note";
@@ -10,6 +10,7 @@ import UsageBadge from "@/components/UsageBadge.vue";
import NoteSweepPane from "@/components/NoteSweepPane.vue";
import StatusBadge from "@/components/StatusBadge.vue";
import PriorityBadge from "@/components/PriorityBadge.vue";
import ScrollSentinel from "@/components/ScrollSentinel.vue";
import GraphView from "@/views/GraphView.vue";
import {
FileText,
@@ -193,7 +194,7 @@ function onClickOutsideNewNote(e: MouseEvent) {
const ID_BATCH = 100; // IDs fetched per server round-trip
const INITIAL_CONTENT = 50; // items loaded on first render
const CONTENT_PAGE = 24; // items loaded per sentinel trigger
const CONTENT_PAGE = 24; // items loaded each time the end of the feed nears
const REFILL_THRESHOLD = 48; // fetch more IDs when queue drops below this
const items = ref<KnowledgeItem[]>([]);
@@ -214,11 +215,21 @@ const totalKnowledge = ref(0); // total matching items on server
const allIdsFetched = ref(false); // no more ID pages to fetch
const idsFetching = ref(false); // ID batch request in-flight
const contentFetching = ref(false); // content batch request in-flight
const sentinelVisible = ref(false); // updated by IntersectionObserver
const moreFailed = ref(false); // the last page asked for did not load
const loading = computed(() => idsFetching.value || contentFetching.value);
let fetchGen = 0; // incremented on each reset to invalidate stale responses
const feedGen = ref(0); // fetchGen, readable by the template
// What the next page is keyed on: the feed it belongs to and how many of its ids
// have been loaded. Ids, not items: a batch can return fewer records than it
// asked for, and the feed must still move on. Empty once every id is loaded.
const nextCursor = computed(() =>
idQueue.value.length > 0 || !allIdsFetched.value
? `${feedGen.value}:${idOffset.value - idQueue.value.length}`
: "",
);
function buildFilterParams(): URLSearchParams {
const p = new URLSearchParams();
@@ -244,7 +255,9 @@ async function fetchIdBatch(gen: number) {
idOffset.value += data.ids.length;
totalKnowledge.value = data.total;
if (!data.has_more) allIdsFetched.value = true;
} catch { /* silent */ }
} catch {
if (gen === fetchGen) moreFailed.value = true;
}
finally { if (gen === fetchGen) idsFetching.value = false; }
}
@@ -262,26 +275,38 @@ async function loadNextContent(gen: number, count: number) {
if (idQueue.value.length < REFILL_THRESHOLD && !allIdsFetched.value && !idsFetching.value) {
fetchIdBatch(gen);
}
} catch { /* silent */ }
finally {
if (gen !== fetchGen) return;
contentFetching.value = false;
// Sentinel may still be visible if new items didn't push it off screen
await nextTick();
if (sentinelVisible.value && idQueue.value.length > 0 && gen === fetchGen) {
loadNextContent(gen, CONTENT_PAGE);
} catch {
// The ids go back to the front of the queue, so Try again asks for the
// same records rather than skipping past them.
if (gen === fetchGen) {
idQueue.value.unshift(...toLoad);
moreFailed.value = true;
}
}
finally {
if (gen === fetchGen) contentFetching.value = false;
}
}
// The next page of the feed: more ids first when the queue has run dry.
async function loadMore() {
const gen = fetchGen;
moreFailed.value = false;
if (idQueue.value.length === 0) await fetchIdBatch(gen);
if (gen !== fetchGen) return;
await loadNextContent(gen, CONTENT_PAGE);
}
async function reset() {
fetchGen++;
feedGen.value = fetchGen;
const gen = fetchGen;
items.value = [];
idQueue.value = [];
idOffset.value = 0;
totalKnowledge.value = 0;
allIdsFetched.value = false;
moreFailed.value = false;
await fetchIdBatch(gen);
if (gen !== fetchGen) return;
await loadNextContent(gen, INITIAL_CONTENT);
@@ -294,23 +319,16 @@ async function fetchTags() {
} catch { /* silent */ }
}
async function resetAndReobserve() {
await reset();
await nextTick();
setupObserver();
}
function onSearchInput() {
if (searchDebounce) clearTimeout(searchDebounce);
searchDebounce = setTimeout(() => resetAndReobserve(), 380);
searchDebounce = setTimeout(() => reset(), 380);
}
watch([activeType, sortMode], () => resetAndReobserve());
// Closing the sweep remounts the feed, and with it the scroll sentinel — a
// fresh element the old observer is not watching. Without this the list loads
// its first page and then never loads another.
watch(sweepActive, (open) => { if (!open) resetAndReobserve(); });
watch(activeTag, () => { fetchCounts(); resetAndReobserve(); });
watch([activeType, sortMode], () => reset());
// Closing the sweep starts the feed over, so a note confirmed in the sweep is
// shown as it now stands.
watch(sweepActive, (open) => { if (!open) reset(); });
watch(activeTag, () => { fetchCounts(); reset(); });
// ─── Today bar ────────────────────────────────────────────────────────────────
@@ -377,24 +395,6 @@ function formatDate(iso: string): string {
return d.toLocaleDateString(undefined, { month: "short", day: "numeric", year: "numeric" });
}
// ─── Sentinel observer ────────────────────────────────────────────────────────
const sentinelEl = ref<HTMLElement | null>(null);
let observer: IntersectionObserver | null = null;
function setupObserver() {
observer?.disconnect();
if (!sentinelEl.value) return;
observer = new IntersectionObserver(
([entry]) => {
sentinelVisible.value = entry.isIntersecting;
if (entry.isIntersecting) loadNextContent(fetchGen, CONTENT_PAGE);
},
{ root: null, rootMargin: "200px" }
);
observer.observe(sentinelEl.value);
}
// ─── Lifecycle ────────────────────────────────────────────────────────────────
onMounted(async () => {
@@ -403,14 +403,11 @@ onMounted(async () => {
fetchTags();
fetchCounts();
fetchTodayBar();
await nextTick();
setupObserver();
});
onUnmounted(() => {
document.removeEventListener('click', onClickOutsideNewNote);
if (searchDebounce) clearTimeout(searchDebounce);
observer?.disconnect();
});
</script>
@@ -583,10 +580,7 @@ onUnmounted(() => {
</div>
<!-- The whole feed stands down while the sweep is open: two answers
to two different questions on one screen is neither. Wrapped
rather than given an extra v-if branch, because the scroll
sentinel lives inside the grid and the observer must not be left
holding a ref to something that never renders. -->
to two different questions on one screen is neither. -->
<template v-if="!sweepActive">
<!-- Loading / empty -->
<div v-if="loading && items.length === 0" class="knowledge-empty">Loading…</div>
@@ -675,10 +669,12 @@ onUnmounted(() => {
</div>
</div>
<!-- Sentinel — IntersectionObserver triggers next content batch -->
<div ref="sentinelEl" class="scroll-sentinel">
<span v-if="contentFetching" class="sentinel-loading">Loading…</span>
</div>
<ScrollSentinel
:cursor="nextCursor"
:loading="loading"
:failed="moreFailed"
@more="loadMore"
/>
</div>
</template>
</div>
@@ -1115,19 +1111,6 @@ onUnmounted(() => {
opacity: 0.85;
}
/* ── Sentinel ────────────────────────────────────────────── */
.scroll-sentinel {
grid-column: 1 / -1;
height: 40px;
display: flex;
align-items: center;
justify-content: center;
}
.sentinel-loading {
font-size: 0.8rem;
color: var(--fs-text-tertiary);
}
/* ── Graph panel ─────────────────────────────────────────── */
.graph-panel {
width: 500px;
+5 -52
View File
@@ -11,6 +11,7 @@ import PaginationBar from "@/components/PaginationBar.vue";
import MomentsSettings from "@/components/MomentsSettings.vue";
import ReplyShapesSettings from "@/components/ReplyShapesSettings.vue";
import BackupScheduleSettings from "@/components/BackupScheduleSettings.vue";
import RestoreBackupDialog from "@/components/RestoreBackupDialog.vue";
import { formatBytes } from "@/utils/bytes";
import { fmtDate, fmtLogStamp } from "@/utils/dateFormat";
import { fetchVersion, type VersionPayload } from "@/api/version";
@@ -467,7 +468,7 @@ const confirmNewPassword = ref("");
const changingPassword = ref(false);
const invalidatingSessions = ref(false);
const exporting = ref(false);
const restoring = ref(false);
const showRestore = ref(false);
// Backup, export and restore walk the whole store, so they are slow BY DESIGN
// and the client's ordinary 30s default would cut them off mid-work. They are
// still bounded — every wait has a deadline, not necessarily a short one — and
@@ -509,7 +510,6 @@ async function copyCommit() {
commitCopied.value = true;
setTimeout(() => { commitCopied.value = false; }, 2000);
}
const restoreFileInput = ref<HTMLInputElement | null>(null);
// Migrate stored "admin" → "config"; unknown tabs fall back to "general"
const VALID_TABS = new Set(["general", "account", "notifications", "integrations", "data", "apikeys", "config", "users", "logs", "groups", "areas", "platforms"]);
@@ -1079,10 +1079,6 @@ async function exportNotes(format: "markdown" | "json") {
}
}
function triggerRestoreUpload() {
restoreFileInput.value?.click();
}
async function saveNotifications() {
savingNotifications.value = true;
notificationsSaved.value = false;
@@ -1279,42 +1275,6 @@ async function runDbMaintenanceNow() {
}
}
async function handleRestoreFile(event: Event) {
const file = (event.target as HTMLInputElement).files?.[0];
if (!file) return;
restoring.value = true;
try {
// The scheduled backup's .tar goes up as it is; the JSON download is
// parsed first so a damaged file fails here, before the upload.
const isTar = file.name.toLowerCase().endsWith(".tar");
const res = await fetch("/api/admin/restore", {
method: "POST",
headers: { "Content-Type": isTar ? "application/x-tar" : "application/json" },
body: isTar ? file : JSON.stringify(JSON.parse(await file.text())),
signal: bulkDeadline(),
});
if (res.status === 413) {
throw new Error(
"the file is bigger than the web upload allows. Restore it with " +
"`python -m grimoire.restore <file>` inside the Grimoire container instead.",
);
}
if (!res.ok) {
const body = await res.json().catch(() => ({ error: `Error ${res.status}` }));
throw new Error((body as Record<string, string>).error || `Error ${res.status}`);
}
const result = await res.json();
toastStore.show(
`Restored ${result.stats?.users ?? 0} users, ${result.stats?.notes ?? 0} notes, ${result.stats?.conversations ?? 0} conversations`
);
} catch (e) {
toastStore.show("Restore failed: " + (e as Error).message, "error");
} finally {
restoring.value = false;
if (restoreFileInput.value) restoreFileInput.value.value = "";
}
}
async function testSearch() {
const q = searchQuery.value.trim();
if (!q) return;
@@ -2429,17 +2389,11 @@ async function deleteUser(userId: number) {
<button class="btn-secondary" @click="exportData('full')" :disabled="exporting">
{{ exporting ? "Exporting..." : "Full Backup" }}
</button>
<button class="btn-secondary btn-warn" @click="triggerRestoreUpload" :disabled="restoring">
{{ restoring ? "Restoring..." : "Restore from Backup" }}
<button class="btn-secondary btn-warn" @click="showRestore = true">
Restore from Backup
</button>
<input
ref="restoreFileInput"
type="file"
accept=".json,.tar"
class="hidden-file-input"
@change="handleRestoreFile"
/>
</div>
<RestoreBackupDialog v-if="showRestore" @close="showRestore = false" />
</section>
<section class="settings-section full-width">
<h2>Scheduled backups</h2>
@@ -3597,7 +3551,6 @@ async function deleteUser(userId: number) {
flex-wrap: wrap;
gap: 0.5rem;
}
.hidden-file-input { display: none; }
/* Checkboxes */
.checkbox-field { margin-bottom: 1rem; }
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "grimoire",
"description": "Grimoire for Claude Code: connects the grimoire MCP server, adds the hooks that deliver live project state and relevant records at the right moment, ships the shared client-neutral Grimoire skills (using-grimoire, writing-plans, reporting-back, systematic-debugging, verification, brainstorming, reusing-code, shape-accounting, family-canon), and syncs your saved Grimoire Processes as skills (/grimoire:sync).",
"version": "2026.10.26.2029",
"version": "2026.11.05.0301",
"author": {
"name": "Bryan Van Deusen"
},
+2 -1
View File
@@ -15,7 +15,7 @@ another one means adding files, not moving or rewriting any.
|---|---|---|
| **The skills** | `plugin/skills/*/SKILL.md` | Agent Skills (the open SKILL.md format). They state every Grimoire reflex in full and name no client. `tests/test_guidance_ownership.py` fails if a skill names a particular client, or references anything outside its own folder. Every client package ships this folder verbatim. |
| **The MCP server** | `<base URL>/mcp` | HTTP, `Authorization: Bearer <fmcp_ key>`. Its `_INSTRUCTIONS` is a client-neutral orientation — the workflow across tools (≤1,600 chars); each tool's description carries its contract; in-band responses (`placement`, `report_back`, `systems_hint`, the duplicate gate, the guessed-id refusal) fire in every client. |
| **The adapter API** | `<base URL>/api/plugin/*` | Endpoints any client's hooks can call with the same key. `GET`, so read scope is enough: `context` (live session state), `retrieve` (rules, preferences and notes for a message), `prior-art` (records and shape-ledger hints for code being written), `tool-rules` (rules for a command about to run), `moment-tools` (which tools reach a mounted moment), `shape-check` (definitions written this turn that nobody has judged), `claim-session` / `release-session` (task claims), `processes` (stored Processes to expose as skills). `POST`, so write scope: `moment` (rules mounted on the moments a tool call reaches), `rule-opened` (a rule was read), `reply-rules` (the finished reply, held for an unopened rule or — when the turn closed a task — for missing completion sections). |
| **The adapter API** | `<base URL>/api/plugin/*` | Endpoints any client's hooks can call with the same key. `GET`, so read scope is enough: `context` (live session state), `retrieve` (rules, preferences and notes for a message), `prior-art` (records and shape-ledger hints for code being written), `tool-rules` (rules for a command about to run), `moment-tools` (which tools reach a mounted moment), `shape-check` (definitions written this turn that nobody has judged), `claim-session` / `release-session` (task claims), `write-scope` (which project a pending write lands in, and whether this session may make it), `processes` (stored Processes to expose as skills). `POST`, so write scope: `moment` (rules mounted on the moments a tool call reaches), `rule-opened` (a rule was read), `reply-rules` (the finished reply, held for an unopened rule or — when the turn closed a task — for missing completion sections). |
| **The API key** | Grimoire → Settings → MCP Access | One `fmcp_` key per install. Write scope covers the MCP tools and every hook; a read key runs only the `GET` hooks. |
## Added by each client
@@ -39,6 +39,7 @@ another one means adding files, not moving or rewriting any.
| `hooks/grimoire_prior_art.sh` | PreToolUse on editor writes: `GET /api/plugin/prior-art`. |
| `hooks/grimoire_after_write.sh` | PostToolUse on shell commands: the same check for code written through the shell. |
| `hooks/grimoire_tool_rules.sh` | PreToolUse on shell commands: `GET /api/plugin/tool-rules`. |
| `hooks/grimoire_project_fence.sh` | PreToolUse on editor writes, shell commands and Grimoire's tools: `GET /api/plugin/write-scope` for an act aimed at another project, answered `ask` with the reason; PostToolUse on `enter_project` remembers the project a session without one entered. |
| `hooks/grimoire_moment.sh` | PreToolUse on every tool: the rules mounted on the moments the call reaches, `POST /api/plugin/moment`; skips tools `GET /api/plugin/moment-tools` says reach nothing mounted, and Grimoire's own (their responses carry `moment_rules`). |
| `hooks/grimoire_reply_check.sh` | Stop: sends the finished reply to `POST /api/plugin/reply-rules` — the rules mounted on the reply moments, and the reply against every rule's trigger, and — when the turn closed a task — the completion-section check; holds once per rule, with the reason the server returns, and never holds the rewrite. |
| `hooks/grimoire_shape_check.sh` | Stop: sends the definitions the turn wrote (the write hooks' `<sid>.written.ids` ledger) to `GET /api/plugin/shape-check`; blocks once, with the reason the server returns, so the agent judges what it built. |
+14
View File
@@ -79,6 +79,20 @@ On install you'll be asked for:
message, as a short menu of titles and matched passages.
- `hooks/hooks.json` → PreToolUse hook on `Bash` (`hooks/grimoire_tool_rules.sh`)
→ `GET /api/plugin/tool-rules`: the rules that match the command about to run.
- `hooks/hooks.json` → PreToolUse hook on editor writes, `Bash` and Grimoire's
tools (`hooks/grimoire_project_fence.sh`) → `GET /api/plugin/write-scope`:
holds a session to its own project. A session belongs to the project its
starting directory resolves to (a `.grimoire` marker, else the git remote's
binding), or, started outside any, to the first project it enters. Editing
another project's files, a git commit, push, merge, rebase, pull, tag or
reset in its repo (from `git -C`, after `cd`, or in a worktree of it), and a
Grimoire write that lands in another project each stop on a permission
prompt naming that project, in every permission mode. Filing an issue there
(`create_task(kind="issue")`) and every read go ahead. An edit in the
session's own repo never calls the server. It can't see a non-git shell
write into another repo (`sed -i`, a redirect), but it fences that repo's
commit and push. When the instance doesn't answer, the act goes ahead and
the hook says so once per outage.
- `hooks/hooks.json` → PreToolUse hook on every tool (`hooks/grimoire_moment.sh`)
→ `POST /api/plugin/moment`: the rules mounted on the moments the call
reaches. It skips tools that `GET /api/plugin/moment-tools` says reach
+246
View File
@@ -0,0 +1,246 @@
#!/usr/bin/env bash
# Grimoire — hold a session to its own project.
#
# A session belongs to the project its starting directory resolves to
# (CLAUDE_PROJECT_DIR: a `.grimoire` marker, else the git remote's binding). A
# session that starts outside any project belongs to the first project it
# enters. Toward any other project a session may read anything and file an
# issue. Every other write stops on a permission prompt naming the other
# project, so the operator decides:
# - editing a file in another project's repo (Write, Edit, MultiEdit,
# NotebookEdit);
# - a git command that changes history or a remote there (commit, push,
# merge, rebase, pull, cherry-pick, revert, tag, am, reset, worktree add),
# wherever it runs from: `git -C DIR`, `cd DIR && git …`, or the shell's
# current directory;
# - a Grimoire write that lands in another project.
#
# `ask`, not `deny`. Which projects a session may change is the operator's
# call, and `ask` puts it to them each time, in every permission mode,
# bypassPermissions included.
#
# The server says where an act lands (GET /api/plugin/write-scope). Two cases
# never reach it:
# - a directory that resolves the same way as the session's own, which is
# nearly every edit;
# - a tool named like a read (READ_PREFIXES in services/write_scope.py, held
# equal to FENCE_READ_PREFIXES below by tests/test_write_scope.py).
#
# What it can't see: a non-git write from the shell (`sed -i`, a redirect)
# into another repo. That repo's commit and push are still fenced.
#
# EXIT 0, ALWAYS. When the server doesn't answer, the act goes ahead and the
# hook says once per outage that it couldn't check.
#
# Run as PostToolUse on enter_project, it records the project a session
# without one has entered, and says nothing.
set -uo pipefail
# shellcheck source=plugin/hooks/grimoire_defs.sh
. "$(dirname "${BASH_SOURCE[0]}")/grimoire_defs.sh"
FENCE_READ_PREFIXES="get_ list_ search find_ enter_project what_might_apply resolve_design_system"
FENCE_GIT_WRITES="commit push merge rebase pull cherry-pick revert tag am reset"
event=$(cat 2>/dev/null || true)
[ -n "$event" ] || exit 0
event_flat=$(printf '%s' "$event" | grimoire_json_flat)
[ -n "$event_flat" ] || exit 0
session_id=$(grimoire_json_pick "$event_flat" '.session_id')
tool_name=$(grimoire_json_pick "$event_flat" '.tool_name')
hook_event=$(grimoire_json_pick "$event_flat" '.hook_event_name')
cwd=$(grimoire_json_pick "$event_flat" '.cwd')
# Its own directory, which no compaction sweeps: the project a session
# entered stays its project for the whole session.
state_dir="${TMPDIR:-/tmp}/grimoire-fence"
safe_sid=$(printf '%s' "${session_id:-none}" | tr -c 'A-Za-z0-9._-' '_')
entered_file="$state_dir/${safe_sid}.entered"
# ── PostToolUse on enter_project: remember the first project entered ─────
if [ "$hook_event" = "PostToolUse" ]; then
case "$tool_name" in
*__enter_project)
pid=$(grimoire_json_pick "$event_flat" '.tool_input.project_id' | tr -cd '0-9')
if [ -n "$pid" ] && [ -n "$session_id" ] && [ ! -s "$entered_file" ]; then
mkdir -p "$state_dir" 2>/dev/null || true
printf '%s' "$pid" > "$entered_file" 2>/dev/null || true
fi
;;
esac
exit 0
fi
# The nearest directory at or above $1 that exists. A Write may create the
# directories its file goes in.
fence_existing_dir() {
local d="$1" depth=0
while [ -n "$d" ] && [ ! -d "$d" ] && [ "$depth" -lt 40 ]; do
d=$(dirname -- "$d" 2>/dev/null) || return 0
depth=$((depth + 1))
done
[ -d "$d" ] && printf '%s' "$d"
}
# "DIR<TAB>SUBCOMMAND" for each git command in the shell command $1 that
# changes history or a remote, with DIR resolved from $2: `cd` moves it for
# the segments after, `git -C` for that one command. Segments split on ; && ||
# | and newlines, and a quoted separator inside a message splits too; the
# words after it are read as a command only if they start with cd or git.
fence_git_targets() {
printf '%s\n' "$1" | awk -v start="$2" -v home="${HOME:-}" -v writes="$FENCE_GIT_WRITES" '
function unq(w) { gsub(/^["\047(]+|["\047)]+$/, "", w); return w }
function at(base, p) {
if (p ~ /^\//) return p
if (p == "~") return home
if (p ~ /^~\//) return home substr(p, 2)
return base "/" p
}
BEGIN { dir = start; n = split(writes, w, " "); for (i = 1; i <= n; i++) iswrite[w[i]] = 1 }
{
line = $0
gsub(/&&|\|\||;|\|/, "\n", line)
m = split(line, segs, "\n")
for (s = 1; s <= m; s++) {
k = split(segs[s], t, /[ \t]+/)
i = 1
while (i <= k && (t[i] == "" || t[i] ~ /^[A-Za-z_][A-Za-z0-9_]*=/)) i++
if (i > k) continue
cmd = unq(t[i])
if (cmd == "cd") {
if (i + 1 <= k && t[i + 1] != "") dir = at(dir, unq(t[i + 1]))
continue
}
if (cmd != "git" && cmd !~ /\/git$/) continue
gdir = dir; i++
while (i <= k && t[i] ~ /^-/) {
if (t[i] == "-C" && i + 1 <= k) { gdir = at(gdir, unq(t[i + 1])); i += 2; continue }
if (t[i] == "-c" || t[i] == "--git-dir" || t[i] == "--work-tree") { i += 2; continue }
i++
}
if (i > k) continue
sub_ = unq(t[i])
if (sub_ == "worktree") { if (i + 1 <= k && unq(t[i + 1]) == "add") print gdir "\tworktree add"; continue }
if (sub_ in iswrite) print gdir "\t" sub_
}
}' 2>/dev/null || true
}
# The short scalar arguments of the MCP call, as a JSON object of strings:
# ids, kind, status, a repo url, and arrays of ids. Long text (bodies, code) is
# left out; nothing that says where a write lands is long.
fence_tool_input() {
printf '%s\n' "$event_flat" | awk -F'\t' '
function add(k, v) { if (!(k in seen)) { seen[k] = 1; order[++n] = k } }
$2 ~ /^\.tool_input\.[A-Za-z_]+$/ && $3 != "null" && length($3) <= 200 {
k = substr($2, 13); add(k); val[k] = "\"" $3 "\""
}
$2 ~ /^\.tool_input\.[A-Za-z_]+\[[0-9]+\]$/ && $3 != "null" && length($3) <= 200 {
k = substr($2, 13); sub(/\[.*$/, "", k); add(k)
# Tested before assigning: mawk creates arr[k] as soon as it is the
# target, so the test inside the assignment would always be true.
item = "\"" $3 "\""
if (k in arr) item = arr[k] "," item
arr[k] = item
}
END {
printf "{"
for (i = 1; i <= n; i++) {
k = order[i]
printf "%s\"%s\":%s", (i > 1 ? "," : ""), k, ((k in arr) ? "[" arr[k] "]" : val[k])
}
printf "}"
}' 2>/dev/null || printf '{}'
}
grimoire_config || exit 0
command -v curl >/dev/null 2>&1 || exit 0
project_dir=${CLAUDE_PROJECT_DIR:-$PWD}
session_scope=$(grimoire_scope_query "$project_dir")
entered=$(cat "$entered_file" 2>/dev/null | tr -cd '0-9')
# No project to hold the session to: started outside any, and entered none.
[ -n "$session_scope" ] || [ -n "$entered" ] || exit 0
# Each check this act needs: "SCOPE<TAB>ACT" for a directory, or a single
# "tool" line for a Grimoire call.
checks=""
case "$tool_name" in
Write|Edit|MultiEdit|NotebookEdit)
path=$(grimoire_json_pick "$event_flat" '.tool_input.file_path')
[ -n "$path" ] || path=$(grimoire_json_pick "$event_flat" '.tool_input.notebook_path')
[ -n "$path" ] || exit 0
case "$path" in /*) ;; *) path="${cwd:-$PWD}/$path" ;; esac
dir=$(fence_existing_dir "$(dirname -- "$path")")
[ -n "$dir" ] || exit 0
scope=$(grimoire_scope_query "$dir")
[ -n "$scope" ] && [ "$scope" != "$session_scope" ] || exit 0
checks=$(printf '%s\t%s' "$scope" "Editing a file in")
;;
Bash)
command=$(grimoire_json_pick "$event_flat" '.tool_input.command')
case "$command" in *git*) ;; *) exit 0 ;; esac
while IFS=$'\t' read -r gdir gsub; do
[ -n "$gdir" ] || continue
[ -d "$gdir" ] || continue
scope=$(grimoire_scope_query "$gdir")
[ -n "$scope" ] && [ "$scope" != "$session_scope" ] || continue
checks="${checks}${checks:+$'\n'}${scope}"$'\t'"git ${gsub} in"
done <<EOF
$(fence_git_targets "$command" "${cwd:-$PWD}")
EOF
[ -n "$checks" ] || exit 0
;;
mcp__*)
short=${tool_name##*__}
for prefix in $FENCE_READ_PREFIXES; do
case "$short" in "$prefix"*) exit 0 ;; esac
done
checks="tool"
;;
*) exit 0 ;;
esac
base="${url%/}/api/plugin/write-scope?"
[ -n "$session_scope" ] && base="${base}${session_scope}&"
[ -n "$entered" ] && base="${base}fallback_project_id=${entered}&"
fence_unreached() {
local marker="$state_dir/${safe_sid}.fence.unreached" now last
mkdir -p "$state_dir" 2>/dev/null || true
now=$(date +%s 2>/dev/null) || now=0
if [ -f "$marker" ]; then
last=$(cat "$marker" 2>/dev/null) || last=0
case "$last" in ''|*[!0-9]*) last=0 ;; esac
[ $((now - last)) -lt 600 ] && return 0
fi
printf '%s' "$now" > "$marker" 2>/dev/null || true
grimoire_json_out "PreToolUse" "> Grimoire did not answer the project fence check, so this act went ahead without being checked against the session's project. If it changes another project's files or records, stop and file an issue there instead. Said once per outage."
}
while IFS= read -r check; do
[ -n "$check" ] || continue
if [ "$check" = "tool" ]; then
input=$(fence_tool_input | grimoire_urlenc)
q="tool=$(printf '%s' "$short" | grimoire_urlenc)&input=${input}"
else
scope=${check%%$'\t'*}
act=${check#*$'\t'}
q="target_${scope}&act=$(printf '%s' "$act" | grimoire_urlenc)"
fi
resp=$(curl -fsS --max-time 4 -H "Authorization: Bearer ${token}" "${base}${q}" 2>/dev/null) || {
fence_unreached
exit 0
}
rm -f "$state_dir/${safe_sid}.fence.unreached" 2>/dev/null || true
flat=$(printf '%s' "$resp" | grimoire_json_flat)
if [ "$(grimoire_json_pick "$flat" '.allowed')" = "false" ]; then
reason=$(grimoire_json_pick "$flat" '.reason')
esc=$(printf '%s' "$reason" | grimoire_json_escape) || exit 0
printf '{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"ask","permissionDecisionReason":"%s"}}\n' "$esc"
exit 0
fi
done <<EOF
$checks
EOF
exit 0
+6 -4
View File
@@ -31,11 +31,13 @@ event_flat=$(printf '%s' "$event" | grimoire_json_flat)
session_id=$(grimoire_json_pick "$event_flat" '.session_id')
[ -n "$session_id" ] || exit 0
# The moment arm's per-session files (its tool cache and the acts ledger)
# describe a session that is over, on any exit — a clear
# included, since the next conversation is a new session id.
# The moment arm's per-session files (its tool cache and the acts ledger) and
# the fence's (the project the session entered, its outage marker) describe a
# session that is over, on any exit — a clear included, since the next
# conversation is a new session id.
safe_sid=$(printf '%s' "$session_id" | tr -c 'A-Za-z0-9._-' '_')
rm -f "${TMPDIR:-/tmp}/grimoire-moment/${safe_sid}".* 2>/dev/null || true
rm -f "${TMPDIR:-/tmp}/grimoire-moment/${safe_sid}".* \
"${TMPDIR:-/tmp}/grimoire-fence/${safe_sid}".* 2>/dev/null || true
reason=$(grimoire_json_pick "$event_flat" '.reason')
[ "$reason" = "clear" ] && exit 0
+18
View File
@@ -25,6 +25,15 @@
}
],
"PreToolUse": [
{
"matcher": "Write|Edit|MultiEdit|NotebookEdit|Bash|mcp__.*grimoire.*",
"hooks": [
{
"type": "command",
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/grimoire_project_fence.sh\""
}
]
},
{
"matcher": "Write|Edit",
"hooks": [
@@ -89,6 +98,15 @@
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/grimoire_claim_session.sh\""
}
]
},
{
"matcher": "mcp__.*__enter_project",
"hooks": [
{
"type": "command",
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/grimoire_project_fence.sh\""
}
]
}
],
"PreCompact": [
+6 -5
View File
@@ -277,11 +277,12 @@ bound — confine the session to it:
- **Pass that `project_id` to every read** (`search`, `list_tasks`,
`list_notes`). An unscoped read bleeds every other project's work into your
context.
- **Only reference or offer work on the in-scope project.** Don't surface,
suggest, or start work on other projects unless the operator explicitly widens
scope.
- If something clearly belongs to a *different* project, say so and **ask before
switching** — never silently operate cross-project.
- **Only reference or offer work on the in-scope project**, unless the
operator explicitly widens scope; work that belongs elsewhere, say so and ask.
- **Cross-project, the one change to make is filing an issue**:
`create_task(project_id=<theirs>, kind="issue")` with how you found it and the
evidence. Reading there is fine; editing its files, committing or pushing in
its repo, or changing its records is the operator's call, and the plugin asks.
## Read these when the moment comes
+17 -5
View File
@@ -395,6 +395,13 @@ SMOKE_EVENTS: dict[str, str] = {
{"session_id": "smoke", "cwd": ".", "hook_event_name": "SessionEnd",
"reason": "prompt_input_exit"}
),
# The project fence. A Grimoire write is the one act that always asks the
# server, so it is what reaches the network in the unreachable case: there
# the act goes ahead and the hook says once that it couldn't check.
"grimoire_project_fence.sh": json.dumps(
{"session_id": "smoke", "cwd": ".", "hook_event_name": "PreToolUse",
"tool_name": "mcp__grimoire__update_task", "tool_input": {"task_id": 1}}
),
# The shared library is sourced, never run; executed bare it defines
# functions and exits — silent by construction.
"grimoire_defs.sh": "",
@@ -415,9 +422,14 @@ STATIC_EMITTERS = {
# would look like a leak; it is the opposite.
"grimoire_precompact_preserve.sh": "preservation instructions present",
}
# The hooks that say so when a configured instance does not answer.
OUTAGE_SPEAKERS = {"grimoire_prior_art.sh", "grimoire_after_write.sh"}
OUTAGE_LINE = "> Grimoire did not answer the prior-art check"
# The hooks that say so when a configured instance does not answer, each with
# the line it says it in.
_PRIOR_ART_OUTAGE = "> Grimoire did not answer the prior-art check"
OUTAGE_LINES = {
"grimoire_prior_art.sh": _PRIOR_ART_OUTAGE,
"grimoire_after_write.sh": _PRIOR_ART_OUTAGE,
"grimoire_project_fence.sh": "> Grimoire did not answer the project fence check",
}
def _run_hook(script: Path, event: str, env_extra: dict[str, str]) -> subprocess.CompletedProcess:
@@ -464,13 +476,13 @@ def check_fail_open() -> None:
f"output must survive having no credentials ({what})")
else:
ok(f"{rel} [{label}]: exit 0, {what}")
elif out and label == "unreachable" and script.name in OUTAGE_SPEAKERS:
elif out and label == "unreachable" and script.name in OUTAGE_LINES:
# The only thing allowed here is the outage line itself.
try:
ctx = json.loads(out)["hookSpecificOutput"]["additionalContext"]
except (ValueError, KeyError, TypeError):
ctx = ""
if ctx.startswith(OUTAGE_LINE):
if ctx.startswith(OUTAGE_LINES[script.name]):
ok(f"{rel} [{label}]: exit 0, says the instance did not answer")
else:
fail(f"{rel} [{label}]: emitted output with no working instance "
+13 -10
View File
@@ -6,7 +6,8 @@ from pathlib import Path
from quart import Quart, g, jsonify, make_response, request, send_from_directory
from grimoire.config import PLACEHOLDER_SECRET_KEYS, SECRET_KEY_FILENAME, Config
from grimoire.config import SECRET_KEY_FILENAME, Config
from grimoire.proxy import PerRequestSecureSession, apply_https_headers
from grimoire.routes.admin import admin_bp
from grimoire.routes.api import api
from grimoire.routes.auth import auth_bp
@@ -61,13 +62,13 @@ def create_app() -> Quart:
app.secret_key = Config.SECRET_KEY
app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
app.config["SESSION_COOKIE_SECURE"] = Config.SECURE_COOKIES
# Secure is decided per request, from the trusted proxy's X-Forwarded-Proto.
app.session_interface = PerRequestSecureSession()
if Config.SECRET_KEY in PLACEHOLDER_SECRET_KEYS:
if os.environ.get("SECURE_COOKIES"):
logger.warning(
"SECRET_KEY is set to a public placeholder, so session cookies are insecure. "
"Remove it to let the instance keep its own key in %s, or set a real one.",
Config.DATA_DIR,
"SECURE_COOKIES is not used: the session cookie is Secure on every request "
"the trusted proxy reports as HTTPS. Remove it from the environment."
)
if Config.SECRET_KEY_SOURCE == "ephemeral":
logger.warning(
@@ -82,11 +83,12 @@ def create_app() -> Quart:
Config.DATA_DIR, SECRET_KEY_FILENAME,
)
if not Config.TRUST_PROXY_HEADERS:
if os.environ.get("TRUST_PROXY_HEADERS") and not os.environ.get("TRUSTED_PROXY_HOPS"):
logger.warning(
"TRUST_PROXY_HEADERS is not set. If this instance is behind a reverse proxy "
"(nginx, Caddy, Traefik) set TRUST_PROXY_HEADERS=true so rate limiting uses "
"real client IPs rather than the proxy IP."
"TRUST_PROXY_HEADERS is replaced by TRUSTED_PROXY_HOPS (now %d). Set "
"TRUSTED_PROXY_HOPS to the number of proxies in front of Grimoire and "
"remove TRUST_PROXY_HEADERS.",
Config.TRUSTED_PROXY_HOPS,
)
app.register_blueprint(admin_bp)
@@ -159,6 +161,7 @@ def create_app() -> Quart:
except Exception:
logger.debug("Failed to log usage", exc_info=True)
apply_https_headers(response)
response.headers.setdefault("X-Content-Type-Options", "nosniff")
response.headers.setdefault("X-Frame-Options", "DENY")
response.headers.setdefault("Referrer-Policy", "strict-origin-when-cross-origin")
+28 -6
View File
@@ -85,6 +85,21 @@ def _secret_key() -> tuple[str, str]:
return secrets.token_hex(32), "ephemeral"
def _trusted_proxy_hops() -> int:
"""TRUSTED_PROXY_HOPS, default 1. The older TRUST_PROXY_HEADERS switch is
still read when the hop count is unset: false means 0 hops. A value that
is not a whole number reads as -1, which validate() refuses."""
raw = os.environ.get("TRUSTED_PROXY_HOPS", "").strip()
if raw:
try:
return int(raw)
except ValueError:
return -1
if os.environ.get("TRUST_PROXY_HEADERS", "").strip().lower() in ("0", "false", "no"):
return 0
return 1
class Config:
DATABASE_URL: str = _read_secret(
"DATABASE_URL",
@@ -95,7 +110,6 @@ class Config:
# shipped compose file, beside the model cache and diagnostics.
DATA_DIR: str = os.environ.get("DATA_DIR", "/data")
SECRET_KEY, SECRET_KEY_SOURCE = _secret_key()
SECURE_COOKIES: bool = os.environ.get("SECURE_COOKIES", "").lower() in ("1", "true", "yes")
LOG_LEVEL: str = os.environ.get("LOG_LEVEL", "INFO")
LOG_RETENTION_DAYS: int = int(os.environ.get("LOG_RETENTION_DAYS", "90"))
@@ -108,7 +122,10 @@ class Config:
SMTP_FROM_NAME: str = os.environ.get("SMTP_FROM_NAME", "Fabled Grimoire")
SMTP_USE_TLS: bool = os.environ.get("SMTP_USE_TLS", "true").lower() in ("1", "true", "yes")
BASE_URL: str = os.environ.get("BASE_URL", "http://localhost:5000").rstrip("/")
TRUST_PROXY_HEADERS: bool = os.environ.get("TRUST_PROXY_HEADERS", "").lower() in ("1", "true", "yes")
# How many reverse proxies stand in front of the app; the client address
# and scheme are read that many entries in from the right of the
# forwarding headers (grimoire/proxy.py). 0 trusts no proxy.
TRUSTED_PROXY_HOPS: int = _trusted_proxy_hops()
# Git URL of the repo that ships this Grimoire plugin. There is one correct
# value per deployment (the repo serving the plugin), so it's a config
@@ -163,11 +180,16 @@ class Config:
errors.append(f"SMTP_PORT={cls.SMTP_PORT} must be between 1 and 65535")
if cls.oidc_enabled() and not cls.BASE_URL.startswith(("http://", "https://")):
errors.append(f"BASE_URL='{cls.BASE_URL}' must start with http:// or https:// when OIDC is enabled")
if cls.SECRET_KEY == "dev-secret-change-me" and cls.SECURE_COOKIES:
if cls.TRUSTED_PROXY_HOPS < 0:
errors.append(
"SECRET_KEY is set to the public placeholder 'dev-secret-change-me' but "
"SECURE_COOKIES=true indicates a production deployment. Remove SECRET_KEY "
"to let the instance keep its own key in DATA_DIR, or set a real one."
f"TRUSTED_PROXY_HOPS={os.environ.get('TRUSTED_PROXY_HOPS')!r} must be a whole "
"number of proxies, 0 or more"
)
if cls.SECRET_KEY in PLACEHOLDER_SECRET_KEYS:
errors.append(
f"SECRET_KEY is set to the public placeholder {cls.SECRET_KEY!r}, so anyone "
"could forge a session. Remove SECRET_KEY to let the instance keep its own "
"key in DATA_DIR, or set a real one."
)
if errors:
raise ValueError("Configuration errors:\n" + "\n".join(f" - {e}" for e in errors))
+1
View File
@@ -94,3 +94,4 @@ from grimoire.models.family import ( # noqa: E402, F401
FamilyAdoption, FamilyDecision, FamilyIdea, FamilyIdeaPlatform,
FamilyIdeaReference, Platform, ProjectPlatform,
)
from grimoire.models.instance import InstanceIdentity # noqa: E402, F401
+26
View File
@@ -0,0 +1,26 @@
from datetime import datetime
from sqlalchemy import CheckConstraint, DateTime, Integer, Text, func
from sqlalchemy.orm import Mapped, mapped_column
from grimoire.models import Base
class InstanceIdentity(Base):
"""The one row naming this database's id lineage.
Two stores with the same instance id number their rows the same way: one
is the other, or a restore of it made with the original ids kept. That is
what lets a backup roll a live instance back by id, and what refuses a
backup whose ids would land on unrelated rows. Made by the migration, so
an instance has it before anyone registers.
"""
__tablename__ = "instance_identity"
__table_args__ = (CheckConstraint("id = 1", name="ck_instance_identity_single_row"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True, default=1)
instance_id: Mapped[str] = mapped_column(Text, nullable=False)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+80
View File
@@ -0,0 +1,80 @@
"""What the trusted reverse proxy says about a request: who sent it, and
whether it arrived over HTTPS.
Grimoire serves plain HTTP; TLS ends at the operator's proxy. The proxy
reports the client's address in X-Forwarded-For and the scheme in
X-Forwarded-Proto, and each proxy on the road APPENDS its entry, so only the
right-hand end of either header was written by a proxy we trust. Anything to
the left of that is whatever the client chose to send.
`TRUSTED_PROXY_HOPS` says how many proxies stand in front of the app (default
1). The entry read is that many in from the right. Zero trusts no proxy: the
headers are ignored and the socket peer is the client.
Never read X-Forwarded-For or X-Forwarded-Proto anywhere else. The client
address keys the rate limits and the audit log, and the scheme decides the
session cookie's Secure flag and HSTS. A second reader is a second trust rule.
"""
from __future__ import annotations
from quart import has_request_context, request
from quart.sessions import SecureCookieSessionInterface
from grimoire.config import Config
# HSTS for a year, sent only on a request the trusted proxy says is HTTPS.
# No includeSubDomains or preload: the app does not own the operator's domain.
HSTS_VALUE = "max-age=31536000"
def trusted_entry(header: str, hops: int) -> str | None:
"""The entry `hops` in from the right of a forwarding header.
A header shorter than that is read at its leftmost entry, the one the
first proxy wrote. A road with fewer proxies than the count (a LAN road
beside a Cloudflare one) still reads right that way. None when no proxy
is trusted or there is no header. Pure, so the trust boundary is
testable without a request."""
if hops <= 0:
return None
entries = [part.strip() for part in header.split(",") if part.strip()]
if not entries:
return None
return entries[-min(hops, len(entries))]
def _forwarded(name: str) -> str | None:
# Joined, because a header sent twice arrives as two values, and each
# proxy's append may land on either one.
return trusted_entry(",".join(request.headers.getlist(name)), Config.TRUSTED_PROXY_HOPS)
def client_ip() -> str:
"""The caller's address, as the trusted proxy saw it."""
if not has_request_context():
return ""
return (_forwarded("X-Forwarded-For") or request.remote_addr or "")[:64]
def is_https() -> bool:
"""Whether this request arrived over HTTPS, by the trusted proxy's word."""
if not has_request_context():
return False
if request.scheme == "https":
return True
return (_forwarded("X-Forwarded-Proto") or "").lower() == "https"
class PerRequestSecureSession(SecureCookieSessionInterface):
"""The session cookie is Secure exactly when the request is HTTPS, so a
public deployment gets it and a plain-HTTP one keeps working."""
def get_cookie_secure(self, app) -> bool:
return is_https()
def apply_https_headers(response) -> None:
"""HSTS on an HTTPS request, and only then. The app never redirects to
HTTPS; plain HTTP is a supported way to run it."""
if is_https():
response.headers.setdefault("Strict-Transport-Security", HSTS_VALUE)
+3 -3
View File
@@ -2,9 +2,9 @@
IMPORTANT — deployment note:
Rate limit counters are stored in memory and are lost on process restart.
When deployed behind a reverse proxy (nginx, Caddy, Traefik) you MUST set
TRUST_PROXY_HEADERS=true so that the real client IP is used as the bucket key
rather than the proxy's IP (which would cause all users to share one bucket).
The bucket key is the client address from grimoire/proxy.py, read by
TRUSTED_PROXY_HOPS. A hop count that is too low keys every caller on the
proxy's address, so they all share one bucket.
"""
import asyncio
+80 -30
View File
@@ -1,20 +1,26 @@
"""Restore a Grimoire backup into an empty instance, from the command line.
"""Restore a Grimoire backup from the command line.
python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar --yes
This is how a fresh instance is brought back after losing the old one. Run it
inside the Grimoire container, which already has the database settings and has
already migrated the schema on start. It reads either the scheduled backup's
tar (services/backup_file.py) or the JSON the admin download produces.
Run it inside the Grimoire container, which already has the database settings
and has migrated the schema on start. It reads the scheduled backup's tar
(services/backup_file.py) or the JSON the admin download produces, and nothing
limits the file's size.
It refuses a database that already holds an account. A restore re-creates
every user, project and record from the file with new ids; into a live
instance it would add a second copy of everything rather than replace it.
(Settings → Backup & Restore still offers the upload for that case, on
purpose and with its own warning.)
It does what Settings → Restore does (services/backup.py replace_from_backup):
- **A new, empty instance** takes the backup as it is.
- **A live instance** rolls back to a backup of its own: everything goes back
to the moment the backup was taken, for every user.
- A backup from another instance, or one made before backups recorded their
instance, only goes into an empty instance.
Unlike the upload, nothing limits the file's size here, so this is also the
way in for a backup too big for the web upload.
Without --yes it only says what would change. With --yes it first saves the
current state as a backup (to the backup destination, or DATA_DIR/pre-restore),
then restores. Everyone is signed out afterwards.
A backup in the oldest (v1) format is copied into an empty instance with new
ids instead, as before.
Afterwards, restart Grimoire: search indexes are not part of a backup and are
rebuilt from the restored records at startup.
@@ -48,9 +54,39 @@ async def _account_count() -> int:
return (await session.execute(select(func.count()).select_from(User))).scalar_one()
async def run(path: Path) -> int:
from grimoire.models import engine
async def _restore_v1(payload: dict) -> int:
from grimoire.services.backup import restore_full_backup
accounts = await _account_count()
if accounts:
print(
f"This database already holds {accounts} account(s). A backup in the oldest "
"format is copied in with new ids rather than replacing anything, so it only "
"goes into an empty instance. Start Grimoire on an empty database and run "
"this again.",
file=sys.stderr,
)
return 3
stats = await restore_full_backup(payload)
print(json.dumps({"restored": stats}, indent=2, default=str))
return 0
def _print_preview(preview: dict) -> None:
taken = preview.get("exported_at") or "an unknown date"
if preview["mode"] == "rollback":
print(f"This rolls the whole instance back to {taken}, for every user. Anything "
"made since is removed, edits since are undone, records deleted since come back.")
else:
print(f"This instance has no records yet; it takes the backup from {taken} as it is.")
for count in preview["counts"]:
print(f" {count['kind']:<24} now {count['now']:>7} after {count['after']:>7}")
async def run(path: Path, *, yes: bool = False) -> int:
from grimoire.models import engine
from grimoire.services import backup_job
from grimoire.services.backup import RestoreRefused, replace_preview
from grimoire.services.backup_file import ArchiveError
try:
@@ -63,20 +99,29 @@ async def run(path: Path) -> int:
print(f"{path} isn't a backup this Grimoire can read: {exc}", file=sys.stderr)
return 2
accounts = await _account_count()
if accounts:
print(
f"This database already holds {accounts} account(s). A restore adds a second "
"copy of everything rather than replacing it, so it only runs into an empty "
"instance. Start Grimoire on an empty database and run this again.",
file=sys.stderr,
)
return 3
stats = await restore_full_backup(payload)
print(json.dumps({"restored": stats}, indent=2, default=str))
print("Restored. Restart Grimoire so the search indexes are rebuilt.", file=sys.stderr)
return 0
if (payload.get("version") or 1) < 2:
code = await _restore_v1(payload)
else:
try:
_print_preview(await replace_preview(payload))
if not yes:
print("Nothing was changed. Run it again with --yes to restore.",
file=sys.stderr)
return 4
result = await backup_job.rollback_to_backup(payload)
except RestoreRefused as exc:
print(str(exc), file=sys.stderr)
return 3
safety = result["safety_backup"]
print(json.dumps({"restored": result["restored"], "safety_backup": safety},
indent=2, default=str))
print(f"The state it replaced is saved as {safety['file']} in {safety['where']}.",
file=sys.stderr)
code = 0
if code == 0:
print("Restored. Restart Grimoire so the search indexes are rebuilt; "
"everyone signs in again.", file=sys.stderr)
return code
finally:
await engine.dispose()
@@ -84,11 +129,16 @@ async def run(path: Path) -> int:
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m grimoire.restore",
description="Restore a Grimoire backup (.tar or .json) into an empty instance.",
description=(
"Restore a Grimoire backup (.tar or .json): into an empty instance, or "
"roll this instance back to a backup of its own."
),
)
parser.add_argument("file", type=Path, help="the backup file")
parser.add_argument("--yes", action="store_true",
help="restore; without it, only say what would change")
args = parser.parse_args(argv)
return asyncio.run(run(args.file))
return asyncio.run(run(args.file, yes=args.yes))
if __name__ == "__main__":
+114 -34
View File
@@ -4,6 +4,7 @@ import json
from quart import Blueprint, Response, g, jsonify, request
from grimoire.auth import admin_required, login_required, get_current_user_id
from grimoire.proxy import client_ip
from grimoire.services.auth import (
create_invitation,
delete_user,
@@ -16,7 +17,6 @@ from grimoire.services.auth import (
from grimoire.services.backup import (
export_full_backup,
export_user_backup,
restore_full_backup,
)
from grimoire.services.email import SMTP_SETTING_KEYS, get_base_url, get_smtp_config, is_smtp_configured, send_test_email
from grimoire.services.logging import get_logs, get_log_stats, log_audit
@@ -46,7 +46,7 @@ async def backup():
else:
data = await export_user_backup(uid)
await log_audit("backup", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details={"scope": scope})
await log_audit("backup", user_id=uid, username=g.user.username, ip_address=client_ip(), details={"scope": scope})
return Response(
json.dumps(data, indent=2, default=str),
content_type="application/json",
@@ -56,31 +56,111 @@ async def backup():
)
async def _restore_payload() -> tuple[dict | None, tuple | None]:
"""The backup a restore request names: `?source=<file>` at the backup
destination, or the uploaded file (the JSON download as
application/json, or the tar as the raw body; held to the request-size
limit, which is why the destination is the way in for a large one).
Returns (payload, None) or (None, an error response)."""
import io
import tempfile
from pathlib import Path
from grimoire.services import backup_destination as dest
from grimoire.services.backup_file import ArchiveError, read_archive
source = request.args.get("source")
try:
if source:
config = await dest.load_config()
secret, _ = await dest.load_secret()
with tempfile.TemporaryDirectory(prefix="grimoire-restore-") as staging:
path = await dest.open_backup(config, secret, source, Path(staging))
def load() -> dict:
with open(path, "rb") as f:
return read_archive(f)
data = await asyncio.to_thread(load)
elif request.mimetype == "application/json":
data = await request.get_json()
else:
data = read_archive(io.BytesIO(await request.get_data()))
except dest.DestinationError as exc:
return None, (jsonify({"error": str(exc)}), 400)
except ArchiveError as exc:
return None, (jsonify({"error": f"That file isn't a Grimoire backup: {exc}"}), 400)
if not isinstance(data, dict) or not data:
return None, (jsonify({"error": "No backup data provided"}), 400)
return data, None
@admin_bp.route("/restore/sources", methods=["GET"])
@admin_required
async def restore_sources():
"""The backups at the backup destination, newest first, to restore from
without uploading one."""
from grimoire.services import backup_destination as dest
config = await dest.load_config()
secret, _ = await dest.load_secret()
try:
backups = await asyncio.wait_for(dest.list_backups(config, secret), _BACKUP_CHECK_TIMEOUT)
except dest.DestinationError as exc:
return jsonify({"backups": [], "error": str(exc)})
except (TimeoutError, asyncio.TimeoutError):
return jsonify({"backups": [], "error": (
f"The destination didn't answer within {int(_BACKUP_CHECK_TIMEOUT)} seconds."
)})
return jsonify({"backups": [{"name": b.name, "size_bytes": b.size_bytes} for b in backups]})
@admin_bp.route("/restore/preview", methods=["POST"])
@admin_required
async def restore_preview():
"""What restoring this backup would do, changing nothing: fresh or
rollback, when it was taken, and the counts now and after. 409 with the
reason when it can't be restored here."""
from grimoire.services.backup import RestoreRefused, replace_preview
data, error = await _restore_payload()
if error:
return error
try:
return jsonify(await replace_preview(data))
except RestoreRefused as exc:
return jsonify({"error": str(exc)}), 409
@admin_bp.route("/restore", methods=["POST"])
@admin_required
async def restore():
"""Restore from an uploaded backup: the JSON download (application/json)
or the scheduled backup's tar (sent as the raw body). The upload is held
to the app's request-size limit; `python -m grimoire.restore` has none, and
is the way in for a bigger file or a fresh instance."""
if request.mimetype == "application/json":
data = await request.get_json()
else:
import io
"""Put the whole instance back to a full backup (services/backup.py
replace_from_backup), after backing up what it replaces. `?confirm=rollback`
is required: this removes everything made since the backup, for every
user. Every session ends, the caller's included; search is rebuilt in the
background."""
from grimoire.services import backup_job as job
from grimoire.services.backup import RestoreRefused
from grimoire.services.backup_file import ArchiveError, read_archive
try:
data = read_archive(io.BytesIO(await request.get_data()))
except ArchiveError as exc:
return jsonify({"error": f"That file isn't a Grimoire backup: {exc}"}), 400
if not data:
return jsonify({"error": "No backup data provided"}), 400
stats = await restore_full_backup(data)
if request.args.get("confirm") != "rollback":
return jsonify({"error": "Restoring replaces the whole instance; confirm it first."}), 400
data, error = await _restore_payload()
if error:
return error
try:
result = await job.rollback_to_backup(data)
except RestoreRefused as exc:
return jsonify({"error": str(exc)}), 409
except job.BackupBusy:
return jsonify({"error": "A backup is running. Restore once it has finished."}), 409
job.start_reindex_after_restore()
uid = get_current_user_id()
await log_audit("restore", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details={"stats": stats})
return jsonify({"status": "ok", "stats": stats})
await log_audit(
"restore", user_id=uid, username=g.user.username, ip_address=client_ip(),
details={"mode": result["mode"], "exported_at": result.get("exported_at"),
"safety_backup": result["safety_backup"], "stats": result["restored"]},
)
return jsonify({"status": "ok", **result})
@admin_bp.route("/users", methods=["GET"])
@@ -100,7 +180,7 @@ async def remove_user(user_id: int):
deleted = await delete_user(user_id)
if not deleted:
return jsonify({"error": "User not found"}), 404
await log_audit("user_delete", user_id=current_uid, username=g.user.username, ip_address=request.remote_addr, details={"deleted_user_id": user_id})
await log_audit("user_delete", user_id=current_uid, username=g.user.username, ip_address=client_ip(), details={"deleted_user_id": user_id})
return jsonify({"status": "ok"})
@@ -121,7 +201,7 @@ async def toggle_registration():
uid = get_current_user_id()
await set_registration_open(uid, bool(open_val))
await log_audit("registration_toggle", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details={"open": bool(open_val)})
await log_audit("registration_toggle", user_id=uid, username=g.user.username, ip_address=client_ip(), details={"open": bool(open_val)})
return jsonify({"status": "ok", "open": bool(open_val)})
@@ -151,7 +231,7 @@ async def update_smtp():
if settings_to_save:
await set_settings_batch(uid, settings_to_save)
await log_audit("smtp_config", user_id=uid, username=g.user.username, ip_address=request.remote_addr)
await log_audit("smtp_config", user_id=uid, username=g.user.username, ip_address=client_ip())
return jsonify({"status": "ok"})
@@ -166,7 +246,7 @@ async def test_smtp():
uid = get_current_user_id()
try:
await send_test_email(recipient)
await log_audit("smtp_test", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details={"recipient": recipient})
await log_audit("smtp_test", user_id=uid, username=g.user.username, ip_address=client_ip(), details={"recipient": recipient})
return jsonify({"status": "ok"})
except Exception as e:
return jsonify({"error": str(e)}), 500
@@ -209,7 +289,7 @@ async def update_forge_webhook_settings():
# The secret is deliberately absent from the audit detail.
await log_audit(
"forge_webhook_config", user_id=uid, username=g.user.username,
ip_address=request.remote_addr, details={},
ip_address=client_ip(), details={},
)
return jsonify({"status": "ok"})
@@ -262,7 +342,7 @@ async def update_base_url():
return jsonify({"error": "Base URL must use http or https"}), 400
uid = get_current_user_id()
await set_setting(uid, "base_url", url)
await log_audit("base_url_config", user_id=uid, username=g.user.username, ip_address=request.remote_addr, details={"base_url": url})
await log_audit("base_url_config", user_id=uid, username=g.user.username, ip_address=client_ip(), details={"base_url": url})
return jsonify({"status": "ok"})
@@ -310,7 +390,7 @@ async def update_db_maintenance():
uid = get_current_user_id()
await log_audit(
"db_maintenance_config", user_id=uid, username=g.user.username,
ip_address=request.remote_addr, details={"enabled": enabled, "hour": hour},
ip_address=client_ip(), details={"enabled": enabled, "hour": hour},
)
return jsonify({"status": "ok", "enabled": enabled, "hour": hour})
@@ -323,7 +403,7 @@ async def run_db_maintenance_now():
uid = get_current_user_id()
await log_audit(
"db_maintenance_run", user_id=uid, username=g.user.username,
ip_address=request.remote_addr,
ip_address=client_ip(),
)
summary = await run_maintenance()
return jsonify(summary)
@@ -389,7 +469,7 @@ async def update_backups():
uid = get_current_user_id()
await log_audit(
"backup_config", user_id=uid, username=g.user.username,
ip_address=request.remote_addr,
ip_address=client_ip(),
details={**config.to_dict(), "secret_changed": bool(
data.get("clear_s3_secret") or (isinstance(secret, str) and secret and secret != SECRET_MASK)
)},
@@ -439,7 +519,7 @@ async def run_backup_now():
return jsonify({"error": "A backup is already running."}), 409
uid = get_current_user_id()
await log_audit("backup_run", user_id=uid, username=g.user.username,
ip_address=request.remote_addr, details={})
ip_address=client_ip(), details={})
return jsonify({"status": "started"}), 202
@@ -463,7 +543,7 @@ async def create_invite():
"invitation_created",
user_id=uid,
username=g.user.username,
ip_address=request.remote_addr,
ip_address=client_ip(),
details={"invited_email": email},
)
return jsonify({"status": "ok"}), 201
@@ -497,7 +577,7 @@ async def delete_invitation(invitation_id: int):
"invitation_revoked",
user_id=uid,
username=g.user.username,
ip_address=request.remote_addr,
ip_address=client_ip(),
details={"invitation_id": invitation_id},
)
return jsonify({"status": "ok"})
+20 -28
View File
@@ -8,6 +8,7 @@ from quart import Blueprint, g, jsonify, redirect, request, session
from grimoire.auth import login_required, get_current_user_id
from grimoire.config import Config
from grimoire.proxy import client_ip
from grimoire.rate_limit import is_rate_limited
from grimoire.services.auth import (
authenticate,
@@ -43,20 +44,11 @@ from grimoire.services.oauth import (
auth_bp = Blueprint("auth", __name__, url_prefix="/api/auth")
def _client_ip() -> str:
if Config.TRUST_PROXY_HEADERS:
for header in ("X-Forwarded-For", "X-Real-IP"):
val = request.headers.get(header, "").split(",")[0].strip()
if val:
return val
return request.remote_addr or ""
@auth_bp.route("/register", methods=["POST"])
async def register():
if not Config.LOCAL_AUTH_ENABLED:
return jsonify({"error": "Local authentication is disabled. Please use SSO."}), 403
if await is_rate_limited(f"register:{_client_ip()}", max_requests=5, window_seconds=300):
if await is_rate_limited(f"register:{client_ip()}", max_requests=5, window_seconds=300):
return jsonify({"error": "Too many registration attempts. Try again later."}), 429
if not await is_registration_open():
return jsonify({"error": "Registration is closed"}), 403
@@ -78,7 +70,7 @@ async def register():
session["user_id"] = user.id
session["session_version"] = user.session_version
await log_audit("register", user_id=user.id, username=user.username, ip_address=_client_ip())
await log_audit("register", user_id=user.id, username=user.username, ip_address=client_ip())
return jsonify(user.to_dict()), 201
@@ -86,7 +78,7 @@ async def register():
async def login():
if not Config.LOCAL_AUTH_ENABLED:
return jsonify({"error": "Local authentication is disabled. Please use SSO."}), 403
if await is_rate_limited(f"login:{_client_ip()}", max_requests=10, window_seconds=60):
if await is_rate_limited(f"login:{client_ip()}", max_requests=10, window_seconds=60):
return jsonify({"error": "Too many login attempts. Try again later."}), 429
data = await request.get_json()
username = (data.get("username") or "").strip()
@@ -97,22 +89,22 @@ async def login():
user = await authenticate(username, password)
if not user:
await log_audit("login_failed", username=username, ip_address=_client_ip())
await log_audit("login_failed", username=username, ip_address=client_ip())
# Try to notify the actual user about the failed attempt
target_user = await get_user_by_username(username)
if target_user:
asyncio.create_task(notify_security_event(
target_user.id, "login_failed",
{"ip_address": _client_ip(), "username": username},
{"ip_address": client_ip(), "username": username},
))
return jsonify({"error": "Invalid username or password"}), 401
session["user_id"] = user.id
session["session_version"] = user.session_version
await log_audit("login", user_id=user.id, username=user.username, ip_address=_client_ip())
await log_audit("login", user_id=user.id, username=user.username, ip_address=client_ip())
asyncio.create_task(notify_security_event(
user.id, "login",
{"ip_address": _client_ip()},
{"ip_address": client_ip()},
))
return jsonify(user.to_dict())
@@ -122,10 +114,10 @@ async def logout():
uid = session.get("user_id")
if uid:
user = await get_user_by_id(uid)
await log_audit("logout", user_id=uid, username=user.username if user else None, ip_address=_client_ip())
await log_audit("logout", user_id=uid, username=user.username if user else None, ip_address=client_ip())
asyncio.create_task(notify_security_event(
uid, "logout",
{"ip_address": _client_ip()},
{"ip_address": client_ip()},
))
session.clear()
return jsonify({"status": "ok"})
@@ -159,10 +151,10 @@ async def update_password():
# Keep the current session alive with the new version (other sessions are now invalidated)
session["session_version"] = new_version
await log_audit("password_change", user_id=uid, username=g.user.username, ip_address=_client_ip())
await log_audit("password_change", user_id=uid, username=g.user.username, ip_address=client_ip())
asyncio.create_task(notify_security_event(
uid, "password_change",
{"ip_address": _client_ip()},
{"ip_address": client_ip()},
))
return jsonify({"status": "ok"})
@@ -178,7 +170,7 @@ async def invalidate_sessions_route():
uid = get_current_user_id()
new_version = await invalidate_other_sessions(uid)
session["session_version"] = new_version
await log_audit("sessions_invalidated", user_id=uid, username=g.user.username, ip_address=_client_ip())
await log_audit("sessions_invalidated", user_id=uid, username=g.user.username, ip_address=client_ip())
return jsonify({"status": "ok"})
@@ -208,14 +200,14 @@ async def update_email():
return jsonify({"error": "Email already in use by another account"}), 409
await update_user_email(uid, new_email)
await log_audit("email_change", user_id=uid, username=user.username, ip_address=_client_ip())
await log_audit("email_change", user_id=uid, username=user.username, ip_address=client_ip())
updated = await get_user_by_id(uid)
return jsonify(updated.to_dict())
@auth_bp.route("/forgot-password", methods=["POST"])
async def forgot_password():
if await is_rate_limited(f"forgot:{_client_ip()}", max_requests=5, window_seconds=300):
if await is_rate_limited(f"forgot:{client_ip()}", max_requests=5, window_seconds=300):
return jsonify({"status": "ok"})
data = await request.get_json()
email = (data.get("email") or "").strip().lower()
@@ -234,7 +226,7 @@ async def forgot_password():
"password_reset_requested",
user_id=user.id,
username=user.username,
ip_address=_client_ip(),
ip_address=client_ip(),
)
return jsonify({"status": "ok"})
@@ -242,7 +234,7 @@ async def forgot_password():
@auth_bp.route("/reset-password", methods=["POST"])
async def reset_password():
if await is_rate_limited(f"reset:{_client_ip()}", max_requests=10, window_seconds=60):
if await is_rate_limited(f"reset:{client_ip()}", max_requests=10, window_seconds=60):
return jsonify({"error": "Too many attempts. Try again later."}), 429
data = await request.get_json()
token = (data.get("token") or "").strip()
@@ -261,7 +253,7 @@ async def reset_password():
if user:
await log_audit(
"password_reset_completed",
user_id=user.id, username=user.username, ip_address=_client_ip(),
user_id=user.id, username=user.username, ip_address=client_ip(),
)
if user.email:
asyncio.create_task(send_password_reset_success_email(user.email))
@@ -304,7 +296,7 @@ async def register_with_invite():
"register_with_invite",
user_id=user.id,
username=user.username,
ip_address=_client_ip(),
ip_address=client_ip(),
)
return jsonify(user.to_dict()), 201
@@ -371,5 +363,5 @@ async def oauth_callback():
user = await find_or_create_oauth_user(sub, email, preferred_username)
session["user_id"] = user.id
session["session_version"] = user.session_version
await log_audit("oauth_login", user_id=user.id, username=user.username, ip_address=_client_ip())
await log_audit("oauth_login", user_id=user.id, username=user.username, ip_address=client_ip())
return redirect("/")
+62
View File
@@ -8,6 +8,8 @@ fmcp_<key>` API key (a read-scoped key suffices for this GET).
"""
from __future__ import annotations
import json
from quart import Blueprint, g, jsonify, request
from grimoire.auth import admin_required, get_current_user_id, login_required
@@ -21,6 +23,8 @@ from grimoire.services import report_check as report_check_svc
from grimoire.services import rule_moment_judgments as judgments_svc
from grimoire.services import shape_check as shape_check_svc
from grimoire.services import task_claims as task_claims_svc
from grimoire.services import write_scope as write_scope_svc
from grimoire.services.access import can_read_project
from grimoire.services.embeddings import memoized_query_embeddings
from grimoire.services.settings import get_admin_setting, set_setting
@@ -626,6 +630,64 @@ async def claim_session():
return jsonify({"claim": claim})
@plugin_bp.get("/write-scope")
@login_required
async def write_scope():
"""May this session do this act? Asked by `grimoire_project_fence.sh`
before a file edit, a git command or a Grimoire write.
A session belongs to the project its starting directory resolves to
(`repo` / `project_id`, as every plugin endpoint reads them), or when
that resolves to none, to `fallback_project_id`: the first project the
session entered. The act is
either a directory (`target_repo` / `target_project_id`) or an MCP call
(`tool`, with `input` as a JSON object of the call's short arguments: the
ids, kind and status that say where it writes). `act` words the act for the
reason, e.g. "git push in".
Returns `{allowed, session_project, targets, issue, reason}` from
services/write_scope.check_act. A GET, so a read-scoped key still fences
file edits and git commands.
"""
session_project, _repo, _unbound = await _project_scope()
if not session_project:
# A session that started outside any project belongs to the first one
# it entered; the hook remembers it.
try:
session_project = int(request.args.get("fallback_project_id") or 0)
except ValueError:
session_project = 0
if session_project and not await can_read_project(g.user.id, session_project):
session_project = 0
tool = (request.args.get("tool") or "").strip()
if tool.startswith("mcp__"):
tool = tool.rsplit("__", 1)[-1]
args: dict = {}
if tool:
try:
parsed = json.loads(request.args.get("input") or "{}")
except ValueError:
return jsonify({"error": "input must be a JSON object"}), 400
if not isinstance(parsed, dict):
return jsonify({"error": "input must be a JSON object"}), 400
args = parsed
target_project = 0
if not tool:
try:
target_project = int(request.args.get("target_project_id") or 0)
except ValueError:
target_project = 0
target_repo = (request.args.get("target_repo") or "").strip()
if not target_project and target_repo:
target_project = await repo_bindings_svc.resolve_project(g.user.id, target_repo) or 0
result = await write_scope_svc.check_act(
g.user.id, session_project, tool=tool, args=args,
target_project=target_project, act=(request.args.get("act") or "")[:80],
)
result["session_project"] = session_project or None
return jsonify(result)
@plugin_bp.get("/release-session")
@login_required
async def release_session():
File diff suppressed because it is too large Load Diff
+63 -3
View File
@@ -4,8 +4,9 @@ Two kinds of destination, both instance-wide and set by an admin:
- **folder**: a directory inside the container, usually a mounted share. A
backup service that watches the share (or a person) takes it from there.
- **s3**: any S3-compatible bucket, addressed by path. Grimoire only puts and
lists, so an append-only key is enough.
- **s3**: any S3-compatible bucket, addressed by path. Grimoire puts and
lists, and reads one back for a restore, so a key that can add and read
objects is enough; it never needs to delete.
Either way Grimoire ADDS one dated file per run and never removes one: keeping
and thinning the history is the destination's job (services/backup_file.py
@@ -28,7 +29,7 @@ from datetime import datetime
from pathlib import Path
from grimoire.services import backup_file
from grimoire.services.backup_s3 import S3Error, S3Target, list_objects, put_file
from grimoire.services.backup_s3 import S3Error, S3Target, get_file, list_objects, put_file
from grimoire.services.secrets import seal, unseal
from grimoire.services.settings import get_admin_setting, set_admin_setting
@@ -260,3 +261,62 @@ async def check_destination(
def _count_backups(folder: Path) -> int:
return sum(1 for name in os.listdir(folder) if backup_file.is_backup_name(name))
@dataclass(frozen=True)
class StoredBackup:
name: str
size_bytes: int
def _taken_order(name: str) -> str:
# The timestamp after the prefix, so grimoire- and scribe- names interleave
# by when they were taken.
return name.split("-", 1)[-1]
async def list_backups(config: BackupConfig, secret: str | None, limit: int = 50) -> list[StoredBackup]:
"""The backups at the destination, newest first."""
if config.destination == "folder":
folder = Path(config.folder_path)
if not folder.is_dir():
raise DestinationError(f"{folder} isn't there, or isn't a folder.")
def scan() -> list[StoredBackup]:
return [
StoredBackup(entry.name, entry.stat().st_size)
for entry in os.scandir(folder)
if entry.is_file() and backup_file.is_backup_name(entry.name)
]
found = await asyncio.to_thread(scan)
else:
problems = config.problems(has_secret=bool(secret))
if problems:
raise DestinationError(" ".join(problems))
try:
objects = await list_objects(s3_target(config, secret or ""))
except S3Error as exc:
raise DestinationError(str(exc)) from None
found = [StoredBackup(o.key, o.size) for o in objects if backup_file.is_backup_name(o.key)]
return sorted(found, key=lambda b: _taken_order(b.name), reverse=True)[:limit]
async def open_backup(config: BackupConfig, secret: str | None, name: str, staging: Path) -> Path:
"""A local path to the backup `name` at the destination: the file itself
in a folder, or a download into `staging` from a bucket. Only a name the
scheduled backup writes is accepted, so the request can't reach any other
file or object."""
if Path(name).name != name or not backup_file.is_backup_name(name):
raise DestinationError(f"{name!r} isn't the name of a Grimoire backup.")
if config.destination == "folder":
path = Path(config.folder_path) / name
if not path.is_file():
raise DestinationError(f"{name} isn't in {config.folder_path}.")
return path
target = s3_target(config, secret or "")
path = staging / name
try:
await get_file(target, name, path)
except S3Error as exc:
raise DestinationError(str(exc)) from None
return path
+66
View File
@@ -24,11 +24,14 @@ import os
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from apscheduler.triggers.cron import CronTrigger
from grimoire.services.backup_destination import (
DestinationError, deliver, load_config, load_secret,
)
from grimoire.services.backup_file import write_archive
from grimoire.services.scheduler import ScheduledJob
from grimoire.services.settings import get_admin_setting, set_admin_setting
@@ -45,6 +48,9 @@ STALE_AFTER = timedelta(hours=26)
# has none, and a run that never returns holds the lock — every later backup
# would be skipped as "already running", silently, forever.
RUN_DEADLINE = timedelta(minutes=60)
# Under DATA_DIR: where a rollback keeps the backup of what it replaced when
# no backup destination takes it.
PRE_RESTORE_DIR = "pre-restore"
_LOCK = asyncio.Lock()
_TASKS: set[asyncio.Task] = set()
@@ -155,6 +161,66 @@ async def _export_and_deliver(config, secret, started):
)
async def save_state_before_restore() -> dict:
"""Back up the store as it is now, before a rollback replaces it, and say
where the file went: the backup destination when backups are on and it
takes the file, otherwise DATA_DIR/pre-restore. Raises when neither takes
it, so a rollback never runs without a way back."""
from grimoire.config import Config
from grimoire.services.backup import export_full_backup
at = datetime.now(timezone.utc)
app_version = os.environ.get("APP_VERSION", "dev")
payload = await export_full_backup()
note = None
config = await load_config()
if config.enabled:
secret, _ = await load_secret()
try:
delivered = await asyncio.wait_for(
deliver(payload, config, secret, at, app_version=app_version),
RUN_DEADLINE.total_seconds(),
)
return {"file": delivered.name, "size_bytes": delivered.size_bytes,
"where": delivered.where}
except (DestinationError, TimeoutError, asyncio.TimeoutError) as exc:
note = (f"The backup destination didn't take it ({exc or 'it timed out'}), "
"so it was kept on the instance instead.")
folder = Path(Config.DATA_DIR) / PRE_RESTORE_DIR
path = await asyncio.to_thread(write_archive, payload, folder, at, app_version=app_version)
saved = {"file": path.name, "size_bytes": path.stat().st_size, "where": str(folder)}
if note:
saved["note"] = note
return saved
async def rollback_to_backup(data: dict) -> dict:
"""Roll the store back to a full backup, after backing up what it
replaces. Refuses (RestoreRefused) before writing anything when the backup
can't be applied here, and raises BackupBusy while a backup is running:
a backup taken during a rollback would mix the two states."""
from grimoire.services.backup import check_replace, replace_from_backup
await check_replace(data)
if _LOCK.locked():
raise BackupBusy()
async with _LOCK:
safety = await save_state_before_restore()
result = await replace_from_backup(data)
result["safety_backup"] = safety
logger.info("rollback: the state it replaced is in %s/%s", safety["where"], safety["file"])
return result
def start_reindex_after_restore() -> None:
"""Rebuild the search vectors a rollback cleared, in the background."""
from grimoire.services.backup import reindex_after_restore
task = asyncio.get_running_loop().create_task(reindex_after_restore())
_TASKS.add(task)
task.add_done_callback(_TASKS.discard)
def start_backup_now() -> None:
"""Start a manual backup in the background (the request returns at once;
the page reads progress from `is_running` and the run list). Raises
+26 -3
View File
@@ -4,9 +4,10 @@ path-style addressing (endpoint/bucket/key), and the two calls a backup needs.
The signing is checked against AWS's published SigV4 examples in
tests/test_services_backup_s3.py.
Grimoire puts and lists; it never deletes. The bucket keeps the history and
thins it on its own policy, so a key that may only add objects is enough —
and is the safer key to hand a service, because a stolen one cannot wipe the
Grimoire puts and lists, and reads a backup back when an admin restores from
the bucket; it never deletes. The bucket keeps the history and thins it on
its own policy, so a key that may only add and read objects is enough — and
is the safer key to hand a service, because a stolen one cannot wipe the
history it writes into.
Written by hand rather than with boto3: two calls don't justify a large
@@ -171,6 +172,28 @@ async def put_file(target: S3Target, key: str, path: Path, *,
raise S3Error(f"Upload of {key} refused: {_says(response)}")
async def get_file(target: S3Target, key: str, path: Path, *,
transport: httpx.AsyncBaseTransport | None = None,
now: datetime | None = None) -> None:
"""Download one object to `path`, streamed to disk."""
url = target.url(key)
headers = sign("GET", url, {}, EMPTY_SHA256, target.access_key, target.secret_key,
target.region, now or datetime.now(timezone.utc))
try:
async with httpx.AsyncClient(verify=target.verify_tls,
timeout=UPLOAD_TIMEOUT_SECONDS,
transport=transport) as client:
async with client.stream("GET", url, headers=headers) as response:
if response.status_code != 200:
await response.aread()
raise S3Error(f"Reading {key} refused: {_says(response)}")
with open(path, "wb") as f:
async for chunk in response.aiter_bytes(CHUNK):
f.write(chunk)
except httpx.HTTPError as exc:
raise _reach_error(target, exc) from None
async def list_objects(target: S3Target, prefix: str = "", limit: int = 1000, *,
transport: httpx.AsyncBaseTransport | None = None,
now: datetime | None = None) -> list[StoredObject]:
+46
View File
@@ -0,0 +1,46 @@
"""This instance's id lineage (models/instance.py says what it means)."""
from __future__ import annotations
import uuid
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
from grimoire.models import async_session
from grimoire.models.instance import InstanceIdentity
async def _read_or_make(session: AsyncSession) -> str:
current = (await session.execute(select(InstanceIdentity.instance_id))).scalar_one_or_none()
if current:
return current
# The migration makes the row; this covers a schema built some other way.
await session.execute(
insert(InstanceIdentity)
.values(id=1, instance_id=str(uuid.uuid4()))
.on_conflict_do_nothing(index_elements=["id"])
)
return (await session.execute(select(InstanceIdentity.instance_id))).scalar_one()
async def get_instance_id(session: AsyncSession | None = None) -> str:
"""This instance's id. Inside `session` when given, so a caller's
transaction sees the same value it may be about to change."""
if session is not None:
return await _read_or_make(session)
async with async_session() as own:
value = await _read_or_make(own)
await own.commit()
return value
async def adopt_instance_id(session: AsyncSession, value: str) -> None:
"""Take a backup's id as this instance's own, inside the caller's
transaction: after a restore that kept the backup's original ids, this
store and the one the backup came from number their rows the same way."""
await session.execute(
insert(InstanceIdentity)
.values(id=1, instance_id=value)
.on_conflict_do_update(index_elements=["id"], set_={"instance_id": value})
)
+338
View File
@@ -0,0 +1,338 @@
"""Which project a write lands in, so a session can be held to its own.
A Claude Code session belongs to one project: the one its starting directory
is bound to. Toward any other project it may read anything and file an issue,
and every other write waits for the operator. The plugin's PreToolUse hook
enforces that, because only the hook knows which session is acting: the MCP
endpoint is stateless and sees nothing but the caller's key. This module
answers the hook's one question: given the session's project and a pending
act, which projects would the act change, and may it go ahead?
An act is either a directory (a file edit or a git command; the hook resolves
it to a repo or a `.grimoire` marker) or an MCP tool call. For a tool call,
`WRITE_TARGETS` says, for every write tool, which of its arguments locate the
write. That means the record or project the write changes, never one it only
mentions: `arose_from_id` on a new task points at another task without
changing it. tests/test_write_scope.py holds the table to the registered tools,
so a new write tool can't be added without saying where it writes.
"""
from __future__ import annotations
from sqlalchemy import or_, select
# How an argument locates the project a write lands in.
PROJECT = "project" # a project id
NOTE = "note" # a task, note, lesson, snippet or process: its project
MILESTONE = "milestone" # a milestone: its project
SYSTEM = "system" # a System: its project
RULE = "rule" # a project rule: its project. A global rule: none
RELATION = "relation" # a rule relation: the projects of its two rules
TRASH = "trash" # a deleted batch: every project its rows belong to
REPO = "repo" # a repo remote: the project it is bound to
DECISION = "decision" # a family decision: its project, else its idea's home
LOCATORS = frozenset({PROJECT, NOTE, MILESTONE, SYSTEM, RULE, RELATION, TRASH, REPO, DECISION})
# Every write tool, and the arguments that locate its write. An empty entry is
# a write that lands in no project:
# - install-level settings (moment mappings, retrieval dials);
# - user-level records (processes, design systems, rulebooks, topics, global
# rules and preferences);
# - judgments the system asks a session for in any project (a rule's outcome,
# a lesson-to-rule link, moment proposals, a menu verdict). Those describe how
# the corpus is retrieved; they don't change another project's work.
WRITE_TARGETS: dict[str, dict[str, str]] = {
# notes, tasks, planning
"create_note": {"project_id": PROJECT},
"update_note": {"note_id": NOTE, "project_id": PROJECT},
"delete_note": {"note_id": NOTE},
"mark_note_verified": {"note_id": NOTE},
"create_lesson": {"project_id": PROJECT},
"update_lesson": {"lesson_id": NOTE},
"delete_lesson": {"lesson_id": NOTE},
"judge_lesson_link": {},
"create_task": {"project_id": PROJECT, "milestone_id": MILESTONE, "parent_id": NOTE},
"update_task": {"task_id": NOTE, "project_id": PROJECT, "milestone_id": MILESTONE},
"delete_task": {"task_id": NOTE},
"add_task_log": {"task_id": NOTE},
"create_records": {"project_id": PROJECT, "milestone_id": MILESTONE},
"start_planning": {"project_id": PROJECT},
"create_milestone": {"project_id": PROJECT},
"update_milestone": {"milestone_id": MILESTONE, "project_id": PROJECT},
"delete_milestone": {"milestone_id": MILESTONE},
# projects, Systems, repos
"create_project": {},
"update_project": {"project_id": PROJECT},
"delete_project": {"project_id": PROJECT},
"decide_project_inception": {"project_id": PROJECT},
"set_project_platforms": {"project_id": PROJECT},
"create_system": {"project_id": PROJECT},
"update_system": {"system_id": SYSTEM},
"delete_system": {"system_id": SYSTEM},
"map_system_to_canonical": {"system_id": SYSTEM},
# Binding a repo changes the project it's bound to and, when it was bound
# elsewhere, the project it leaves.
"bind_repo": {"project_id": PROJECT, "repo_url": REPO},
"unbind_repo": {"repo_url": REPO},
# family canon. An idea is a record in its home project; an adoption is a
# project's answer; a conflict is between the two projects it names.
"propose_family_idea": {"note_id": NOTE},
"promote_family_idea": {"note_id": NOTE},
"retire_family_idea": {"note_id": NOTE},
"revise_family_idea": {"note_id": NOTE},
"set_family_references": {"note_id": NOTE},
"set_family_topic": {"note_id": NOTE},
"undo_family_decision": {"decision_id": DECISION},
"assess_family_adoption": {"project_id": PROJECT},
"resolve_family_conflict": {"canon_project_id": PROJECT, "other_project_id": PROJECT},
# snippets, processes, the shape ledger
"create_snippet": {"project_id": PROJECT},
"update_snippet": {"snippet_id": NOTE, "project_id": PROJECT},
"delete_snippet": {"snippet_id": NOTE},
"verify_snippet": {"snippet_id": NOTE},
"merge_snippets": {"target_id": NOTE, "source_ids": NOTE},
"unmerge_snippet": {"survivor_id": NOTE, "source_id": NOTE},
"create_process": {},
"update_process": {"process_id": NOTE},
"delete_process": {"process_id": NOTE},
"classify_shapes": {"project_id": PROJECT},
"classify_shapes_by_rule": {"project_id": PROJECT},
"confirm_shape_proposals": {"project_id": PROJECT},
"refresh_pattern_coverage": {"project_id": PROJECT},
"move_path": {"project_id": PROJECT},
# design systems belong to their owner, not to a project; choosing one for
# a project is that project's write.
"create_design_system": {},
"update_design_system": {},
"delete_design_system": {},
"create_design_token": {},
"update_design_token": {},
"delete_design_token": {},
"set_project_design_system": {"project_id": PROJECT},
# rules
"create_rulebook": {},
"update_rulebook": {},
"delete_rulebook": {},
"create_topic": {},
"update_topic": {},
"delete_topic": {},
"create_rule": {},
"create_preference": {},
"create_project_rule": {"project_id": PROJECT},
"update_rule": {"rule_id": RULE},
"update_preference": {"rule_id": RULE},
"move_rule": {"rule_id": RULE, "project_id": PROJECT},
"delete_rule": {"rule_id": RULE},
"mark_rule_verified": {"rule_id": RULE},
"relate_rules": {"from_rule_id": RULE, "to_rule_id": RULE},
"unrelate_rules": {"relation_id": RELATION},
"rule_outcome": {},
# moments and retrieval: install-level, and judgments asked for anywhere
"tune_retrieval": {},
"migrate_retrieval_floor": {},
"map_action": {},
"unmap_action": {},
"propose_rule_moments": {},
"judge_rule_moments": {},
"rule_misfired": {},
"judge_menu": {},
# trash: a batch can hold rows from several projects
"restore": {"deleted_batch_id": TRASH},
"purge_trash": {"deleted_batch_id": TRASH},
}
# Argument names that would locate a write wherever they appear. The guard test
# requires every write tool parameter with one of these names to be declared
# above, or listed in REFERENCES below, so a tool that gains one can't write
# past the fence unnoticed.
LOCATOR_NAMES = frozenset({
"project_id", "task_id", "note_id", "lesson_id", "snippet_id", "process_id",
"milestone_id", "system_id", "rule_id", "deleted_batch_id",
})
# Arguments with a locating name that only point at a record, with why each one
# doesn't say where the write lands.
REFERENCES: dict[str, dict[str, str]] = {
"classify_shapes_by_rule": {
"snippet_id": "the canon the shapes are judged against; the rows written are project_id's",
},
"confirm_shape_proposals": {
"snippet_id": "narrows which proposals to confirm; the rows written are project_id's",
},
"judge_lesson_link": {
"lesson_id": "a judgment on a lesson-to-rule pairing the system asked for",
"rule_id": "a judgment on a lesson-to-rule pairing the system asked for",
},
"rule_outcome": {
"rule_id": "telemetry on what a rule changed in this session's own work",
},
"rule_misfired": {
"rule_id": "telemetry on a moment that fired a rule in this session",
"project_id": "the project the misfire happened in, recorded with the report",
},
}
# Tool-name prefixes the plugin's fence answers without asking the server: every
# tool named this way reads. tests/test_write_scope.py holds the hook's copy of
# this list equal to it, and holds every write tool to a name outside it.
READ_PREFIXES = (
"get_", "list_", "search", "find_", "enter_project", "what_might_apply",
"resolve_design_system",
)
# The one write a session may make in another project: filing an issue there,
# as a new open task outside any plan of theirs.
ISSUE_STATUSES = frozenset({"", "todo"})
def is_issue_filing(tool: str, args: dict) -> bool:
"""True for create_task(kind="issue") that opens a fresh, unplanned issue."""
if tool != "create_task":
return False
if str(args.get("kind") or "") != "issue":
return False
if str(args.get("status") or "") not in ISSUE_STATUSES:
return False
return not _positive_ints(args.get("milestone_id")) and not _positive_ints(args.get("parent_id"))
def _positive_ints(value) -> list[int]:
"""The ids an argument names. 0 and negative values mean "none" or "leave
unchanged" across the tools (update_task's project_id=-1 clears), so only
positive ids locate anything."""
values = value if isinstance(value, (list, tuple)) else [value]
out = []
for v in values:
try:
n = int(v)
except (TypeError, ValueError):
continue
if n > 0:
out.append(n)
return out
async def _projects_of(user_id: int, kind: str, value) -> set[int]:
"""The projects the argument `value` of locator `kind` lands in."""
from grimoire.models import async_session
from grimoire.models.family import FamilyDecision
from grimoire.models.milestone import Milestone
from grimoire.models.note import Note
from grimoire.models.rulebook import Rule, RuleRelation
from grimoire.models.system import System
from grimoire.services import repo_bindings
if kind == REPO:
found = await repo_bindings.resolve_project(user_id, str(value or ""))
return {found} if found else set()
if kind == TRASH:
batch = str(value or "").strip()
if not batch:
return set()
async with async_session() as session:
found: set[int] = set()
for model in (Note, Milestone, System, Rule):
rows = await session.execute(
select(model.project_id).where(model.deleted_batch_id == batch)
)
found.update(p for p in rows.scalars() if p)
return found
ids = _positive_ints(value)
if not ids:
return set()
if kind == PROJECT:
return set(ids)
async with async_session() as session:
if kind == RELATION:
rows = await session.execute(
select(Rule.project_id)
.join(RuleRelation, or_(RuleRelation.from_rule_id == Rule.id,
RuleRelation.to_rule_id == Rule.id))
.where(RuleRelation.id.in_(ids))
)
return {p for p in rows.scalars() if p}
if kind == DECISION:
rows = await session.execute(
select(FamilyDecision.project_id, Note.project_id)
.join(Note, Note.id == FamilyDecision.idea_id)
.where(FamilyDecision.id.in_(ids))
)
return {own or home for own, home in rows.all() if own or home}
model = {NOTE: Note, MILESTONE: Milestone, SYSTEM: System, RULE: Rule}[kind]
rows = await session.execute(select(model.project_id).where(model.id.in_(ids)))
return {p for p in rows.scalars() if p}
async def tool_targets(user_id: int, tool: str, args: dict) -> set[int]:
"""Every project the MCP call `tool(**args)` would change. Empty for a
read, a write that lands in no project, and a tool this server doesn't
have."""
found: set[int] = set()
for name, kind in WRITE_TARGETS.get(tool, {}).items():
if name in args:
found |= await _projects_of(user_id, kind, args[name])
return found
async def _titles(ids: list[int]) -> dict[int, str]:
from grimoire.models import async_session
from grimoire.models.project import Project
if not ids:
return {}
async with async_session() as session:
rows = await session.execute(select(Project.id, Project.title).where(Project.id.in_(ids)))
return {pid: title for pid, title in rows.all()}
def _named(project_id: int, titles: dict[int, str]) -> str:
title = titles.get(project_id)
return f"{title} (project {project_id})" if title else f"project {project_id}"
async def check_act(
user_id: int, session_project: int, *, tool: str = "", args: dict | None = None,
target_project: int = 0, act: str = "",
) -> dict:
"""May a session in `session_project` do this act?
The act is an MCP call (`tool` with `args`) or a directory already
resolved to `target_project`; `act` words it for the reason ("git push
in", "Editing a file in"). Returns:
- `allowed`: False when the act would change another project and isn't the
one cross-project write allowed;
- `targets`: the other projects, as {id, title};
- `issue`: the act is that allowed write, filing an issue;
- `reason`: what the operator reads in the prompt, and the session after.
A session with no project is held to nothing. A target the caller can't
read is left out: the write itself is refused there, so there is nothing
to ask about, and naming it would tell the caller where a record lives.
"""
from grimoire.services.access import can_read_project
args = args or {}
if tool:
landing = await tool_targets(user_id, tool, args)
else:
landing = {target_project} if target_project else set()
others = [
p for p in sorted(landing)
if p != session_project and await can_read_project(user_id, p)
]
if not session_project or not others:
return {"allowed": True, "targets": [], "issue": False, "reason": ""}
titles = await _titles([session_project, *others])
targets = [{"id": p, "title": titles.get(p, "")} for p in others]
if tool and is_issue_filing(tool, args):
return {"allowed": True, "targets": targets, "issue": True, "reason": ""}
there = ", ".join(_named(p, titles) for p in others)
what = act.strip() or (f"{tool} changes" if tool else "This changes")
first = others[0]
reason = (
f"This session belongs to {_named(session_project, titles)}. {what} {there}, "
f"a different project. From here, the one change to make there is filing an "
f"issue: create_task(project_id={first}, kind=\"issue\") with how it was found "
f"and the evidence. Approve only if this session should change {there} itself."
)
return {"allowed": False, "targets": targets, "issue": False, "reason": reason}
@@ -5,7 +5,8 @@ the real `restore_full_backup`.
The unit lane already shows the tar hands back the dict it was given. What
only this lane shows is that the dict it hands back is one the restore
accepts — every id remapped to the restored rows, nothing dropped — and that
the command-line restore refuses a database that already has accounts.
the command-line restore refuses, on a live instance, a backup that doesn't
record this instance as its source.
"""
import pytest
import pytest_asyncio
@@ -97,10 +98,11 @@ async def test_the_tar_restores_with_every_id_remapped(written):
assert note.status == "todo"
async def test_the_command_refuses_a_database_with_accounts(written):
"""The source user exists, so this database is not empty: the command
must stop before restoring anything."""
assert await run(written) == 3
async def test_the_command_refuses_a_backup_from_no_known_instance(written):
"""This database is live and the backup names no instance it came from,
so it can't be rolled back to: the command stops before restoring
anything, even with --yes."""
assert await run(written, yes=True) == 3
async with async_session() as s:
restored = (await s.execute(select(User).where(User.username == RESTORED))).scalars().all()
assert restored == []
+214
View File
@@ -0,0 +1,214 @@
"""Real-Postgres proof that a full backup rolls this instance back by id.
The lane shares one database, so every test here takes its backup of the
whole store, changes its own rows, and rolls back to that backup: what it
puts back is exactly what was there when it started.
"""
from datetime import datetime, timezone
from unittest.mock import AsyncMock, patch
import pytest
import pytest_asyncio
from sqlalchemy import func, select
from grimoire.models import async_session
from grimoire.models.note import Note
from grimoire.models.project import Project
from grimoire.models.user import User
from grimoire.restore import run
from grimoire.services import backup, backup_job
from grimoire.services.api_keys import create_api_key, lookup_key
from grimoire.services.backup_file import write_archive
from grimoire.services.instance import adopt_instance_id, get_instance_id
from tests.helpers import ensure_user
pytestmark = [pytest.mark.integration, pytest.mark.usefixtures("_dispose_engine")]
OWNER = "rollback_owner"
@pytest_asyncio.fixture
async def seeded():
"""One user with a project and two tasks, committed, and a full backup
taken right after."""
async with async_session() as s:
user = await ensure_user(s, OWNER)
# A clean slate for this user: an earlier test's rows would otherwise
# be in this test's backup too.
for model in (Note, Project):
for row in (await s.execute(select(model).where(model.user_id == user.id))).scalars():
await s.delete(row)
await s.flush()
project = Project(user_id=user.id, title="Rollback drill")
s.add(project)
await s.flush()
kept = Note(user_id=user.id, project_id=project.id, title="before",
body="as backed up", note_type="note", task_kind="work", status="todo")
doomed = Note(user_id=user.id, project_id=project.id, title="deleted later",
body="", note_type="note", task_kind="work", status="todo")
s.add_all([kept, doomed])
await s.commit()
ids = {"user": user.id, "project": project.id, "kept": kept.id, "doomed": doomed.id}
key, _ = await create_api_key(ids["user"], "rollback drill", "read")
payload = await backup.export_full_backup()
return ids, payload, key
def _own(payload: dict, table: str, user_id: int) -> list[dict]:
return sorted(
(row for row in payload[table] if row.get("user_id") == user_id),
key=lambda row: row["id"],
)
async def _change_everything(ids: dict) -> int:
"""Edit one task, delete the other, add a task and a project. Returns the
new task's id."""
async with async_session() as s:
kept = await s.get(Note, ids["kept"])
kept.title = "after"
kept.body = "edited since the backup"
await s.delete(await s.get(Note, ids["doomed"]))
extra = Project(user_id=ids["user"], title="made after the backup")
s.add(extra)
await s.flush()
new = Note(user_id=ids["user"], project_id=extra.id, title="new since",
body="", note_type="note", task_kind="work", status="todo")
s.add(new)
await s.commit()
return new.id
async def test_a_rollback_puts_back_edits_and_deletions_and_removes_new_records(seeded):
ids, payload, _ = seeded
new_id = await _change_everything(ids)
result = await backup.replace_from_backup(payload)
assert result["mode"] == "rollback"
async with async_session() as s:
kept = await s.get(Note, ids["kept"])
assert (kept.title, kept.body) == ("before", "as backed up")
assert await s.get(Note, ids["doomed"]) is not None
assert await s.get(Note, new_id) is None
titles = (await s.execute(
select(Project.title).where(Project.user_id == ids["user"])
)).scalars().all()
assert titles == ["Rollback drill"]
after = await backup.export_full_backup()
for table in ("projects", "milestones", "notes"):
assert _own(after, table, ids["user"]) == _own(payload, table, ids["user"]), table
async def test_a_surviving_users_api_key_still_works_and_their_sessions_end(seeded):
ids, payload, key = seeded
async with async_session() as s:
version_before = (await s.get(User, ids["user"])).session_version
await _change_everything(ids)
await backup.replace_from_backup(payload)
found = await lookup_key(key)
assert found is not None and found.user_id == ids["user"]
async with async_session() as s:
assert (await s.get(User, ids["user"])).session_version == version_before + 1
async def test_new_records_after_a_rollback_take_fresh_ids(seeded):
ids, payload, _ = seeded
await _change_everything(ids)
await backup.replace_from_backup(payload)
async with async_session() as s:
highest = (await s.execute(select(func.max(Note.id)))).scalar_one()
note = Note(user_id=ids["user"], project_id=ids["project"], title="after rollback",
body="", note_type="note", task_kind="work", status="todo")
s.add(note)
await s.commit()
assert note.id > highest
await s.delete(note)
await s.commit()
@pytest.mark.parametrize("instance_id", ["another-instance", None])
async def test_a_backup_from_elsewhere_is_refused_on_a_live_instance(seeded, instance_id):
ids, payload, _ = seeded
foreign = {**payload, "instance_id": instance_id}
new_id = await _change_everything(ids)
with pytest.raises(backup.RestoreRefused):
await backup.replace_from_backup(foreign)
async with async_session() as s:
assert (await s.get(Note, ids["kept"])).title == "after"
assert await s.get(Note, new_id) is not None
await backup.replace_from_backup(payload)
async def test_a_users_own_export_is_refused(seeded):
_, payload, _ = seeded
with pytest.raises(backup.RestoreRefused):
await backup.replace_from_backup({**payload, "scope": "user"})
async def test_a_failure_part_way_changes_nothing(seeded):
ids, payload, _ = seeded
new_id = await _change_everything(ids)
with patch.object(backup, "_reset_sequences", AsyncMock(side_effect=RuntimeError("boom"))):
with pytest.raises(RuntimeError):
await backup.replace_from_backup(payload)
async with async_session() as s:
assert (await s.get(Note, ids["kept"])).title == "after"
assert await s.get(Note, ids["doomed"]) is None
assert await s.get(Note, new_id) is not None
await backup.replace_from_backup(payload)
async def test_a_fresh_instance_adopts_the_backups_instance_id(seeded):
_, payload, _ = seeded
own = await get_instance_id()
try:
with patch.object(backup, "_is_fresh", AsyncMock(return_value=True)):
result = await backup.replace_from_backup({**payload, "instance_id": "a-lineage"})
assert result["mode"] == "fresh"
assert await get_instance_id() == "a-lineage"
finally:
async with async_session() as s:
await adopt_instance_id(s, own)
await s.commit()
async def test_the_command_only_previews_without_yes(seeded, tmp_path):
ids, payload, _ = seeded
path = write_archive(payload, tmp_path, datetime.now(timezone.utc), app_version="test")
new_id = await _change_everything(ids)
assert await run(path) == 4
async with async_session() as s:
assert (await s.get(Note, ids["kept"])).title == "after"
assert await s.get(Note, new_id) is not None
await backup.replace_from_backup(payload)
async def test_the_command_rolls_back_from_the_tar_with_yes(seeded, tmp_path):
"""The scheduled backup's own file, read back through the command: the
whole store returns to it, after the safety backup is taken."""
ids, payload, _ = seeded
path = write_archive(payload, tmp_path, datetime.now(timezone.utc), app_version="test")
new_id = await _change_everything(ids)
saved = {"file": "grimoire-safety.tar", "size_bytes": 1, "where": str(tmp_path)}
with patch.object(backup_job, "save_state_before_restore",
AsyncMock(return_value=saved)) as save:
assert await run(path, yes=True) == 0
save.assert_awaited_once()
async with async_session() as s:
kept = await s.get(Note, ids["kept"])
assert (kept.title, kept.body) == ("before", "as backed up")
assert await s.get(Note, ids["doomed"]) is not None
assert await s.get(Note, new_id) is None
@@ -0,0 +1,42 @@
"""Real-Postgres proof that every instance has one id lineage, that a full
backup carries it, and that adopting a backup's id replaces this one's."""
import io
import pytest
from grimoire.models import async_session
from grimoire.services import backup
from grimoire.services.backup_file import read_archive, write_archive_to
from grimoire.services.instance import adopt_instance_id, get_instance_id
pytestmark = [pytest.mark.integration, pytest.mark.usefixtures("_dispose_engine")]
async def test_the_migration_gave_this_instance_an_id_and_it_stays_put():
first = await get_instance_id()
assert first and len(first) >= 32
assert await get_instance_id() == first
async def test_a_full_backup_carries_the_instance_id_through_the_tar():
payload = await backup.export_full_backup()
assert payload["instance_id"] == await get_instance_id()
out = io.BytesIO()
write_archive_to(payload, out, app_version="test")
out.seek(0)
assert read_archive(out)["instance_id"] == payload["instance_id"]
async def test_adopting_a_backups_id_replaces_this_instances():
own = await get_instance_id()
try:
async with async_session() as s:
await adopt_instance_id(s, "adopted-lineage")
await s.commit()
assert await get_instance_id() == "adopted-lineage"
finally:
async with async_session() as s:
await adopt_instance_id(s, own)
await s.commit()
assert await get_instance_id() == own
+86
View File
@@ -0,0 +1,86 @@
"""Real-Postgres proof that a write is traced to the project it lands in.
Two projects of one user: the session's own and another. Each locator kind is
resolved from a real row, and check_act answers for a session standing in
the first.
"""
import pytest
import pytest_asyncio
from sqlalchemy import select
from grimoire.models import async_session
from grimoire.models.milestone import Milestone
from grimoire.models.note import Note
from grimoire.models.project import Project
from grimoire.models.system import System
from grimoire.services import write_scope
from tests.helpers import ensure_user
pytestmark = [pytest.mark.integration, pytest.mark.usefixtures("_dispose_engine")]
OWNER = "write_scope_owner"
BATCH = "write-scope-test-batch"
@pytest_asyncio.fixture
async def two_projects():
async with async_session() as s:
user = await ensure_user(s, OWNER)
for model in (Note, Milestone, System, Project):
for row in (await s.execute(select(model).where(model.user_id == user.id))).scalars():
await s.delete(row)
await s.flush()
home = Project(user_id=user.id, title="Home")
other = Project(user_id=user.id, title="Elsewhere")
s.add_all([home, other])
await s.flush()
plan = Milestone(user_id=user.id, project_id=other.id, title="Their plan")
area = System(user_id=user.id, project_id=other.id, name="Their area")
s.add_all([plan, area])
await s.flush()
theirs = Note(user_id=user.id, project_id=other.id, milestone_id=plan.id,
title="Their task", body="", note_type="note", task_kind="work",
status="todo")
mine = Note(user_id=user.id, project_id=home.id, title="My task", body="",
note_type="note", task_kind="work", status="todo")
trashed = Note(user_id=user.id, project_id=other.id, title="Trashed", body="",
note_type="note", deleted_batch_id=BATCH)
s.add_all([theirs, mine, trashed])
await s.commit()
return {"user": user.id, "home": home.id, "other": other.id, "plan": plan.id,
"area": area.id, "theirs": theirs.id, "mine": mine.id}
async def test_each_locator_finds_the_project_of_the_row_it_names(two_projects):
ids = two_projects
other, home, uid = ids["other"], ids["home"], ids["user"]
async def lands(tool, **args):
return await write_scope.tool_targets(uid, tool, args)
assert await lands("add_task_log", task_id=ids["theirs"]) == {other}
assert await lands("update_task", task_id=ids["mine"]) == {home}
assert await lands("update_task", task_id=ids["mine"], milestone_id=ids["plan"]) == {home, other}
assert await lands("delete_milestone", milestone_id=ids["plan"]) == {other}
assert await lands("update_system", system_id=ids["area"]) == {other}
assert await lands("restore", deleted_batch_id=BATCH) == {other}
assert await lands("create_rule", topic_id=1) == set()
async def test_a_session_in_one_project_is_asked_before_writing_in_the_other(two_projects):
ids = two_projects
uid, home, other = ids["user"], ids["home"], ids["other"]
own = await write_scope.check_act(uid, home, tool="add_task_log",
args={"task_id": ids["mine"]})
assert own["allowed"] is True
theirs = await write_scope.check_act(uid, home, tool="add_task_log",
args={"task_id": ids["theirs"]})
assert theirs["allowed"] is False
assert theirs["targets"] == [{"id": other, "title": "Elsewhere"}]
assert "Home" in theirs["reason"] and "Elsewhere" in theirs["reason"]
issue = await write_scope.check_act(uid, home, tool="create_task",
args={"project_id": other, "kind": "issue"})
assert issue["allowed"] is True and issue["issue"] is True
+214
View File
@@ -0,0 +1,214 @@
"""The project fence, end to end: plugin/hooks/grimoire_project_fence.sh run
through a real shell against real git repos and a stand-in server.
A session started in one repo edits, commits and pushes in another, and makes
Grimoire writes. Each act aimed elsewhere is asked about in the server's
words; acts in the session's own repo never reach the server.
"""
from __future__ import annotations
import json
import os
import subprocess
from pathlib import Path
from tests.helpers import http_sink, need_tools
HOOK = Path(__file__).resolve().parents[1] / "plugin" / "hooks" / "grimoire_project_fence.sh"
PATH = "/api/plugin/write-scope"
HOME_URL = "https://forge.example/team/home.git"
OTHER_URL = "https://forge.example/team/other.git"
ASK = json.dumps({"allowed": False, "targets": [{"id": 9, "title": "Other"}], "issue": False,
"reason": "This session belongs to Home. Editing a file in Other, a different project."}).encode()
OK = b'{"allowed": true, "targets": [], "issue": false, "reason": ""}'
def _git(*args, cwd):
subprocess.run(["git", *args], cwd=cwd, check=True, capture_output=True,
env={**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@example.com",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@example.com"})
def _repo(tmp_path: Path, name: str, remote: str) -> Path:
path = tmp_path / name
path.mkdir()
_git("init", "-q", cwd=path)
_git("remote", "add", "origin", remote, cwd=path)
return path
def _run(tmp_path, port, event, project_dir):
need_tools("bash", "curl", "awk", "git")
env = {"PATH": os.environ["PATH"], "GRIMOIRE_URL": f"http://127.0.0.1:{port}",
"GRIMOIRE_TOKEN": "t", "TMPDIR": str(tmp_path / "tmp"), "HOME": str(tmp_path),
"CLAUDE_PROJECT_DIR": str(project_dir)}
(tmp_path / "tmp").mkdir(exist_ok=True)
event = {"session_id": "s-fence", "hook_event_name": "PreToolUse", **event}
out = subprocess.run(["bash", str(HOOK)], input=json.dumps(event), capture_output=True,
text=True, env=env, timeout=30)
assert out.returncode == 0, out.stderr
return out.stdout.strip()
def _asked(out: str) -> str:
decision = json.loads(out)["hookSpecificOutput"]
assert decision["permissionDecision"] == "ask"
return decision["permissionDecisionReason"]
def _edit(path, cwd):
return {"tool_name": "Edit", "cwd": str(cwd),
"tool_input": {"file_path": str(path), "old_string": "a", "new_string": "b"}}
def _bash(command, cwd):
return {"tool_name": "Bash", "cwd": str(cwd), "tool_input": {"command": command}}
def test_an_edit_in_the_sessions_own_repo_never_asks_the_server(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _edit(home / "src" / "a.py", home), home)
assert out == "" and seen == []
def test_an_edit_in_another_projects_repo_is_asked_about_in_the_servers_words(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _edit(other / "new_dir" / "b.py", home), home)
assert "a different project" in _asked(out)
q = seen[0]
assert q["_path"] == PATH
assert q["repo"] == [HOME_URL] and q["target_repo"] == [OTHER_URL]
assert q["act"] == ["Editing a file in"]
def test_an_edit_the_server_allows_goes_ahead_silently(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
with http_sink(by_path={PATH: OK}) as (port, seen):
out = _run(tmp_path, port, _edit(other / "b.py", home), home)
assert out == "" and len(seen) == 1
def test_an_edit_outside_every_repo_is_no_projects_business(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
loose = tmp_path / "loose"
loose.mkdir()
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _edit(loose / "notes.txt", home), home)
assert out == "" and seen == []
def test_a_commit_through_git_dash_c_is_checked_against_that_repo(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
command = f'git -C {other} commit -m "tidy && push later"'
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _bash(command, home), home)
_asked(out)
assert seen[0]["target_repo"] == [OTHER_URL]
assert seen[0]["act"] == ["git commit in"]
def test_a_push_after_cd_is_checked_against_the_repo_it_moved_to(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
_repo(tmp_path, "other", OTHER_URL)
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _bash("cd ../other && git push origin dev", home), home)
_asked(out)
assert seen[0]["target_repo"] == [OTHER_URL] and seen[0]["act"] == ["git push in"]
def test_git_commands_that_change_nothing_and_writes_at_home_stay_local(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
with http_sink(by_path={PATH: ASK}) as (port, seen):
assert _run(tmp_path, port, _bash(f"git -C {other} status && git -C {other} log -1", home), home) == ""
assert _run(tmp_path, port, _bash("git commit -m x && git push", home), home) == ""
assert _run(tmp_path, port, _bash("echo hello", home), home) == ""
assert seen == []
def test_a_worktree_of_another_repo_belongs_to_that_repo(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
_git("commit", "-q", "--allow-empty", "-m", "start", cwd=other)
tree = tmp_path / "other-tree"
_git("worktree", "add", "-q", str(tree), cwd=other)
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, _edit(tree / "c.py", home), home)
_asked(out)
assert seen[0]["target_repo"] == [OTHER_URL]
def test_a_grimoire_write_sends_its_short_arguments_and_is_asked_about(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
event = {"tool_name": "mcp__plugin_grimoire_grimoire__update_task", "cwd": str(home),
"tool_input": {"task_id": 5, "status": "done", "body": "x" * 500,
"system_ids": [3, 4]}}
with http_sink(by_path={PATH: ASK}) as (port, seen):
out = _run(tmp_path, port, event, home)
_asked(out)
q = seen[0]
assert q["tool"] == ["update_task"]
assert json.loads(q["input"][0]) == {"task_id": "5", "status": "done", "system_ids": ["3", "4"]}
def test_a_grimoire_read_never_asks_the_server(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
event = {"tool_name": "mcp__plugin_grimoire_grimoire__get_task", "cwd": str(home),
"tool_input": {"task_id": 5}}
with http_sink(by_path={PATH: ASK}) as (port, seen):
assert _run(tmp_path, port, event, home) == ""
assert seen == []
def test_a_session_started_outside_any_project_takes_the_first_one_it_enters(tmp_path):
loose = tmp_path / "loose"
loose.mkdir()
write = {"tool_name": "mcp__plugin_grimoire_grimoire__add_task_log", "cwd": str(loose),
"tool_input": {"task_id": 5, "content": "c"}}
with http_sink(by_path={PATH: ASK}) as (port, seen):
# Nothing to hold the session to yet.
assert _run(tmp_path, port, write, loose) == ""
assert seen == []
for pid in (7, 8):
_run(tmp_path, port, {"hook_event_name": "PostToolUse", "cwd": str(loose),
"tool_name": "mcp__plugin_grimoire_grimoire__enter_project",
"tool_input": {"project_id": pid}}, loose)
_asked(_run(tmp_path, port, write, loose))
assert seen[0]["fallback_project_id"] == ["7"]
assert "repo" not in seen[0]
def test_an_unanswered_check_lets_the_act_through_and_says_so_once(tmp_path):
home = _repo(tmp_path, "home", HOME_URL)
other = _repo(tmp_path, "other", OTHER_URL)
first = _run(tmp_path, 1, _edit(other / "b.py", home), home)
ctx = json.loads(first)["hookSpecificOutput"]["additionalContext"]
assert ctx.startswith("> Grimoire did not answer the project fence check")
assert _run(tmp_path, 1, _edit(other / "b.py", home), home) == ""
def test_the_entered_project_outlives_a_compaction_and_ends_with_the_session(tmp_path):
# A compaction loses the context, not the session: the project it entered
# stays its project. Session end clears it.
loose = tmp_path / "loose"
loose.mkdir()
_run(tmp_path, 1, {"hook_event_name": "PostToolUse", "cwd": str(loose),
"tool_name": "mcp__plugin_grimoire_grimoire__enter_project",
"tool_input": {"project_id": 7}}, loose)
entered = tmp_path / "tmp" / "grimoire-fence" / "s-fence.entered"
assert entered.read_text() == "7"
env = {"PATH": os.environ["PATH"], "TMPDIR": str(tmp_path / "tmp"), "HOME": str(tmp_path)}
hooks = HOOK.parent
for hook, event in (("grimoire_session_context.sh", {"source": "compact"}),
("grimoire_session_end.sh", {"reason": "other"})):
subprocess.run(["bash", str(hooks / hook)], input=json.dumps({"session_id": "s-fence", **event}),
capture_output=True, text=True, env=env, timeout=60, check=True)
if hook == "grimoire_session_context.sh":
assert entered.exists(), "a compaction cleared the project the session entered"
assert not entered.exists()
+186
View File
@@ -0,0 +1,186 @@
"""The trusted proxy decides the client address and whether a request is HTTPS.
Only the entry TRUSTED_PROXY_HOPS in from the right of a forwarding header was
written by a proxy the app trusts; a client can put anything to its left. The
session cookie's Secure flag and HSTS follow the trusted X-Forwarded-Proto, and
plain HTTP is left untouched.
"""
import pytest
from quart import Quart, Response, session
from grimoire import config, proxy
from grimoire.config import Config
from grimoire.proxy import (
HSTS_VALUE, PerRequestSecureSession, apply_https_headers, client_ip, is_https,
trusted_entry,
)
CLIENT = "203.0.113.7"
FORGED = "198.51.100.1"
PEER = "192.0.2.10"
@pytest.mark.parametrize("header, hops, expected", [
("", 1, None),
(CLIENT, 0, None),
(CLIENT, 1, CLIENT),
(f"{FORGED}, {CLIENT}", 1, CLIENT),
(f"{FORGED}, {CLIENT}, {PEER}", 2, CLIENT),
# Shorter than the hop count: the leftmost entry, written by the first proxy.
(CLIENT, 2, CLIENT),
(" , ", 1, None),
])
def test_trusted_entry_counts_in_from_the_right(header, hops, expected):
assert trusted_entry(header, hops) == expected
def _app() -> Quart:
app = Quart(__name__)
app.secret_key = "test-only-key"
app.session_interface = PerRequestSecureSession()
@app.route("/")
async def index():
session["seen"] = True
response = Response("ok")
apply_https_headers(response)
return response
return app
def _context(app, headers=None, scheme="http"):
# Quart's ASGI layer writes the socket peer into a Remote-Addr header,
# replacing any the client sent; a test request context takes it as given.
return app.test_request_context(
"/", headers={"Remote-Addr": PEER, **(headers or {})}, scheme=scheme,
)
async def test_a_client_forged_address_left_of_the_proxys_is_ignored(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 1)
async with _context(_app(), {"X-Forwarded-For": f"{FORGED}, {CLIENT}"}):
assert client_ip() == CLIENT
async def test_zero_hops_ignores_the_header_and_reads_the_socket_peer(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 0)
async with _context(_app(), {"X-Forwarded-For": CLIENT}):
assert client_ip() == PEER
async def test_no_header_reads_the_socket_peer(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 1)
async with _context(_app()):
assert client_ip() == PEER
async def test_a_header_sent_twice_is_read_as_one_chain(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 1)
headers = [("Remote-Addr", PEER), ("X-Forwarded-For", FORGED), ("X-Forwarded-For", CLIENT)]
async with _app().test_request_context("/", headers=headers):
assert client_ip() == CLIENT
async def test_a_short_proto_header_is_read_where_the_first_proxy_wrote_it(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 2)
async with _context(_app(), {"X-Forwarded-Proto": "https"}):
assert is_https()
async def test_a_forged_https_left_of_the_proxys_http_is_ignored(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 1)
async with _context(_app(), {"X-Forwarded-Proto": "https, http"}):
assert not is_https()
async def test_a_direct_https_request_is_https_at_any_hop_count(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", 0)
async with _context(_app(), scheme="https"):
assert is_https()
def test_outside_a_request_nothing_is_trusted():
assert client_ip() == ""
assert not is_https()
async def _get(monkeypatch, hops, headers):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", hops)
response = await _app().test_client().get("/", headers=headers)
cookie = " ".join(response.headers.getlist("Set-Cookie"))
assert "session=" in cookie
return cookie, response.headers.get("Strict-Transport-Security")
async def test_hops_zero_with_https_header_sends_no_secure_flag_and_no_hsts(monkeypatch):
cookie, hsts = await _get(monkeypatch, 0, {"X-Forwarded-Proto": "https"})
assert "Secure" not in cookie
assert hsts is None
async def test_hops_one_with_https_header_sends_secure_flag_and_hsts(monkeypatch):
cookie, hsts = await _get(monkeypatch, 1, {"X-Forwarded-Proto": "https"})
assert "Secure" in cookie
assert hsts == HSTS_VALUE
async def test_a_plain_request_sends_neither(monkeypatch):
cookie, hsts = await _get(monkeypatch, 1, {})
assert "Secure" not in cookie
assert hsts is None
def test_the_app_decides_secure_per_request():
from grimoire.app import create_app
assert isinstance(create_app().session_interface, PerRequestSecureSession)
def test_the_app_reads_forwarding_headers_only_through_this_module():
"""A second reader of X-Forwarded-* or the socket peer would be a second
trust rule. Matches the reads themselves, not prose that names a header."""
import pathlib
import re
read = re.compile(
r"""headers\.(?:get|getlist|__getitem__)\(\s*["']X-(?:Forwarded|Real)"""
r"""|headers\[\s*["']X-(?:Forwarded|Real)"""
r"""|\.remote_addr\b""",
re.IGNORECASE,
)
src = pathlib.Path(proxy.__file__).parent
readers = sorted(
str(path.relative_to(src)) for path in src.rglob("*.py")
if path.name != "proxy.py" and read.search(path.read_text())
)
assert readers == []
assert read.search((src / "proxy.py").read_text()), "the pattern no longer finds the real reader"
@pytest.mark.parametrize("env, expected", [
({}, 1),
({"TRUSTED_PROXY_HOPS": "2"}, 2),
({"TRUSTED_PROXY_HOPS": "0"}, 0),
({"TRUSTED_PROXY_HOPS": "two"}, -1),
({"TRUST_PROXY_HEADERS": "false"}, 0),
({"TRUST_PROXY_HEADERS": "true"}, 1),
({"TRUST_PROXY_HEADERS": "false", "TRUSTED_PROXY_HOPS": "1"}, 1),
])
def test_the_hop_count_from_the_environment(monkeypatch, env, expected):
monkeypatch.delenv("TRUSTED_PROXY_HOPS", raising=False)
monkeypatch.delenv("TRUST_PROXY_HEADERS", raising=False)
for name, value in env.items():
monkeypatch.setenv(name, value)
assert config._trusted_proxy_hops() == expected
def test_a_hop_count_that_is_not_a_number_refuses_to_start(monkeypatch):
monkeypatch.setattr(Config, "TRUSTED_PROXY_HOPS", -1)
with pytest.raises(ValueError, match="TRUSTED_PROXY_HOPS"):
Config.validate()
@pytest.mark.parametrize("key", sorted(config.PLACEHOLDER_SECRET_KEYS))
def test_a_placeholder_secret_key_refuses_to_start(monkeypatch, key):
monkeypatch.setattr(Config, "SECRET_KEY", key)
with pytest.raises(ValueError, match="public placeholder"):
Config.validate()
+6
View File
@@ -52,3 +52,9 @@ def test_service_surface_and_allowlist():
def test_set_admin_setting_exists():
from grimoire.services.settings import set_admin_setting
assert callable(set_admin_setting)
def test_restore_routes_registered():
from grimoire.app import create_app
rules = {r.rule for r in create_app().url_map.iter_rules()}
assert {"/api/admin/restore", "/api/admin/restore/preview", "/api/admin/restore/sources"} <= rules
+45 -3
View File
@@ -11,7 +11,7 @@ The full FK-remapping round-trip needs real Postgres and lives in the
"""
from datetime import datetime, timezone
from types import SimpleNamespace
from unittest.mock import patch
from unittest.mock import AsyncMock, patch
import pytest
@@ -22,7 +22,7 @@ def test_backup_version_is_current():
"""The bump is the point of the test — a payload section added without
moving the version produces backups that are structurally different and
indistinguishable by inspection."""
assert backup.BACKUP_VERSION == 27
assert backup.BACKUP_VERSION == 28
def _exportable_note(**over):
@@ -561,11 +561,15 @@ class _CM:
@pytest.mark.asyncio
async def test_export_full_backup_contains_every_declared_section():
with patch("grimoire.services.backup.async_session", lambda: _CM()):
with patch("grimoire.services.backup.async_session", lambda: _CM()), patch(
"grimoire.services.instance.get_instance_id", AsyncMock(return_value="test-lineage"),
):
out = await backup.export_full_backup()
assert out["version"] == backup.BACKUP_VERSION
assert out["scope"] == "full"
# v28: the id lineage a live rollback is checked against.
assert out["instance_id"] == "test-lineage"
assert "api_keys" in out["_not_included"]
# The sections v2 silently dropped, the six v5 added, v6's
# note_supersessions, and v7's code_shapes (all empty here).
@@ -681,3 +685,41 @@ def test_rule_rows_carry_the_kind_so_a_preference_does_not_restore_as_a_rule():
arose_from_id=None, created_at=stamp, updated_at=stamp,
)
assert backup._rule_rows([row])[0]["kind"] == "preference"
def test_a_rollback_never_cascades_into_a_row_it_keeps_for_outside_references():
"""A rollback replaces most backed-up tables whole, and keeps an identity
table's rows in place so what points at them from OUTSIDE the backup (API
keys, shares, forge connections) survives. If such a kept row had an
ON DELETE CASCADE foreign key into a table replaced whole, clearing that
table would delete the kept row, and its keys and shares with it."""
from grimoire.models import Base
tables = Base.metadata.tables
backed_up = set(backup._BACKED_UP)
referenced_from_outside = {
fk.column.table.name
for table in Base.metadata.sorted_tables if table.name not in backed_up
for fk in table.foreign_keys if fk.column.table.name in backed_up
}
assert {"users", "projects", "notes"} <= referenced_from_outside
assert referenced_from_outside <= set(backup._IDENTITY_TABLES)
safe_targets = set(backup._IDENTITY_TABLES) | set(backup._CATALOG_TABLES) | (
set(tables) - backed_up
)
unsafe = sorted(
f"{name}.{fk.parent.name} -> {fk.column.table.name}"
for name in referenced_from_outside
for fk in tables[name].foreign_keys
if (fk.ondelete or "").upper() == "CASCADE" and fk.column.table.name not in safe_targets
)
assert unsafe == []
def test_every_rollback_table_list_names_real_backed_up_tables():
from grimoire.models import Base
assert set(backup._IDENTITY_TABLES) <= set(backup._BACKED_UP)
assert set(backup._CATALOG_TABLES) <= set(backup._BACKED_UP)
assert set(backup._DERIVED_TABLES) <= set(backup._NOT_INCLUDED)
assert set(backup._DERIVED_TABLES) <= set(Base.metadata.tables)
+44
View File
@@ -188,3 +188,47 @@ def test_checking_s3_only_lists():
said = asyncio.run(check_destination(S3, "secret", transport=httpx.MockTransport(handler)))
assert methods == ["GET", "GET"]
assert "holds 2 Grimoire backup" in said
# ── Restoring from what is already at the destination ──────────────────────
def test_a_folder_lists_its_backups_newest_first_and_nothing_else(tmp_path):
for name in ("grimoire-20261012T020000Z.tar", "scribe-20261011T020000Z.tar",
"grimoire-20261013T020000Z.tar", "notes.txt", ".grimoire-20261014T020000Z.tar.part"):
(tmp_path / name).write_bytes(b"x" * 10)
found = asyncio.run(backup_destination.list_backups(BackupConfig(folder_path=str(tmp_path)), None))
assert [b.name for b in found] == [
"grimoire-20261013T020000Z.tar", "grimoire-20261012T020000Z.tar",
"scribe-20261011T020000Z.tar",
]
assert found[0].size_bytes == 10
def test_a_missing_folder_says_so_when_listing(tmp_path):
with pytest.raises(DestinationError, match="isn't there"):
asyncio.run(backup_destination.list_backups(
BackupConfig(folder_path=str(tmp_path / "absent")), None))
def test_opening_a_folder_backup_is_the_file_itself(tmp_path):
(tmp_path / "grimoire-20261012T020000Z.tar").write_bytes(b"x")
path = asyncio.run(backup_destination.open_backup(
BackupConfig(folder_path=str(tmp_path)), None, "grimoire-20261012T020000Z.tar", tmp_path))
assert path == tmp_path / "grimoire-20261012T020000Z.tar"
@pytest.mark.parametrize("name", [
"../etc/passwd", "grimoire-x/../../secret.tar", "secret_key", "/data/secret_key",
"grimoire-20261012T020000Z.json",
])
def test_only_a_backup_name_can_be_opened(tmp_path, name):
with pytest.raises(DestinationError, match="isn't the name of a Grimoire backup"):
asyncio.run(backup_destination.open_backup(
BackupConfig(folder_path=str(tmp_path)), None, name, tmp_path))
def test_a_backup_not_in_the_folder_says_so(tmp_path):
with pytest.raises(DestinationError, match="isn't in"):
asyncio.run(backup_destination.open_backup(
BackupConfig(folder_path=str(tmp_path)), None, "grimoire-20261012T020000Z.tar", tmp_path))
+121
View File
@@ -221,3 +221,124 @@ def test_routes_and_scheduler_are_wired():
assert callable(backup_job.start_backup_scheduler)
assert callable(backup_job.stop_backup_scheduler)
assert backup_destination.SECRET_SETTING == "backup_s3_secret"
# ── The backup a rollback takes of what it replaces ────────────────────────
def _safety_patches(tmp_path, *, config, deliver=None):
from grimoire.config import Config
async def load_config():
return config
async def load_secret():
return (None, False)
async def export():
return {"version": 28, "scope": "full", "users": []}
async def delivered(payload, config, secret, at, **kw):
return Delivered("grimoire-20261015T020000Z.tar", 1234, "/backups")
return [
patch.object(backup_job, "load_config", load_config),
patch.object(backup_job, "load_secret", load_secret),
patch.object(backup_job, "deliver", deliver or delivered),
patch("grimoire.services.backup.export_full_backup", export),
patch.object(Config, "DATA_DIR", str(tmp_path)),
]
def test_the_state_before_a_rollback_goes_to_the_backup_destination(tmp_path):
saved = _run(backup_job.save_state_before_restore(),
_safety_patches(tmp_path, config=BackupConfig(enabled=True)))
assert saved == {"file": "grimoire-20261015T020000Z.tar", "size_bytes": 1234,
"where": "/backups"}
assert not (tmp_path / backup_job.PRE_RESTORE_DIR).exists()
def test_with_backups_off_it_is_kept_on_the_instance(tmp_path):
saved = _run(backup_job.save_state_before_restore(),
_safety_patches(tmp_path, config=BackupConfig(enabled=False)))
folder = tmp_path / backup_job.PRE_RESTORE_DIR
assert saved["where"] == str(folder)
assert (folder / saved["file"]).stat().st_size == saved["size_bytes"] > 0
assert "note" not in saved
def test_a_destination_that_refuses_it_falls_back_to_the_instance_and_says_so(tmp_path):
async def refuse(*a, **kw):
raise DestinationError("/backups isn't there.")
saved = _run(backup_job.save_state_before_restore(),
_safety_patches(tmp_path, config=BackupConfig(enabled=True), deliver=refuse))
assert saved["where"] == str(tmp_path / backup_job.PRE_RESTORE_DIR)
assert "/backups isn't there." in saved["note"]
def test_a_rollback_that_cannot_save_the_current_state_does_not_run(tmp_path):
from grimoire.services import backup
calls = []
async def check(data):
return "rollback"
async def replace(data):
calls.append("replace")
return {"mode": "rollback"}
async def fail():
raise OSError("disk full")
patches = [
patch.object(backup, "check_replace", check),
patch.object(backup, "replace_from_backup", replace),
patch.object(backup_job, "save_state_before_restore", fail),
]
with pytest.raises(OSError):
_run(backup_job.rollback_to_backup({}), patches)
assert calls == []
def test_a_refused_rollback_writes_no_safety_backup():
from grimoire.services import backup
calls = []
async def check(data):
raise backup.RestoreRefused("from another instance")
async def save():
calls.append("save")
patches = [
patch.object(backup, "check_replace", check),
patch.object(backup_job, "save_state_before_restore", save),
]
with pytest.raises(backup.RestoreRefused):
_run(backup_job.rollback_to_backup({}), patches)
assert calls == []
def test_a_rollback_reports_where_the_state_it_replaced_went():
from grimoire.services import backup
async def check(data):
return "rollback"
async def replace(data):
return {"mode": "rollback", "restored": {"notes": 3}}
async def save():
return {"file": "grimoire-x.tar", "size_bytes": 9, "where": "/data/pre-restore"}
patches = [
patch.object(backup, "check_replace", check),
patch.object(backup, "replace_from_backup", replace),
patch.object(backup_job, "save_state_before_restore", save),
]
result = _run(backup_job.rollback_to_backup({}), patches)
assert result["safety_backup"]["where"] == "/data/pre-restore"
assert result["restored"] == {"notes": 3}
+24
View File
@@ -126,3 +126,27 @@ def test_list_reads_contents():
s3.StoredObject("grimoire-20261011T000000Z.tar", 1200, "2026-10-11T00:00:01.000Z"),
s3.StoredObject("grimoire-20261012T000000Z.tar", 1300, "2026-10-12T00:00:01.000Z"),
]
def test_get_streams_the_object_to_disk(tmp_path):
seen = {}
def handler(request):
seen["method"], seen["url"] = request.method, str(request.url)
seen["sha"] = request.headers["x-amz-content-sha256"]
return httpx.Response(200, content=b"y" * 3000)
out = tmp_path / "b.tar"
asyncio.run(s3.get_file(TARGET, "b.tar", out, transport=httpx.MockTransport(handler)))
assert seen["method"] == "GET"
assert seen["url"] == "http://archives:8081/s3/grimoire/b.tar"
assert seen["sha"] == s3.EMPTY_SHA256
assert out.read_bytes() == b"y" * 3000
def test_a_refused_get_says_so_and_writes_nothing(tmp_path):
transport = httpx.MockTransport(lambda r: httpx.Response(403, content=b"<Error><Code>AccessDenied</Code></Error>"))
out = tmp_path / "b.tar"
with pytest.raises(s3.S3Error, match="b.tar"):
asyncio.run(s3.get_file(TARGET, "b.tar", out, transport=transport))
assert not out.exists()
+250
View File
@@ -0,0 +1,250 @@
"""Where a write lands, so the plugin can hold a session to its own project.
The guards here keep services/write_scope.WRITE_TARGETS whole: every write
tool says which of its arguments locate the write, and an argument that would
locate one can't be left out. The rest covers the one cross-project write
allowed (filing an issue) and the answer check_act gives.
"""
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import pytest
from quart import g
from grimoire.services import write_scope
from grimoire.services.write_scope import (
LOCATOR_NAMES, LOCATORS, REFERENCES, WRITE_TARGETS, check_act, is_issue_filing,
)
def _write_tool_params() -> dict[str, set[str]]:
from grimoire.mcp.server import _WRITE_TOOLS, build_mcp_server
manager = build_mcp_server()._tool_manager
return {
name: set(manager.get_tool(name).parameters.get("properties", {}))
for name in _WRITE_TOOLS
}
def test_every_write_tool_says_where_it_writes():
from grimoire.mcp.server import _WRITE_TOOLS
missing = sorted(_WRITE_TOOLS - set(WRITE_TARGETS))
assert not missing, (
f"write tools with no entry in WRITE_TARGETS: {missing}. Say which "
"arguments locate each one's write, or {} when it lands in no project."
)
extra = sorted(set(WRITE_TARGETS) - _WRITE_TOOLS)
assert not extra, f"WRITE_TARGETS names tools that aren't write tools: {extra}"
def test_every_declared_argument_is_one_the_tool_takes_and_has_a_known_kind():
params = _write_tool_params()
for tool, locators in WRITE_TARGETS.items():
for arg, kind in locators.items():
assert arg in params[tool], f"{tool} has no argument {arg!r}"
assert kind in LOCATORS, f"{tool}.{arg}: unknown locator {kind!r}"
def test_an_argument_that_would_locate_a_write_cannot_be_left_out():
params = _write_tool_params()
undeclared = sorted(
f"{tool}.{arg}"
for tool, names in params.items()
for arg in names & LOCATOR_NAMES
if arg not in WRITE_TARGETS[tool] and arg not in REFERENCES.get(tool, {})
)
assert not undeclared, (
f"write tools take these locating arguments without declaring them in "
f"WRITE_TARGETS, or saying in REFERENCES why they only point: {undeclared}"
)
def test_a_reference_is_an_argument_the_tool_takes_and_not_also_a_locator():
params = _write_tool_params()
for tool, refs in REFERENCES.items():
for arg in refs:
assert arg in params[tool], f"REFERENCES names {tool}.{arg}, which it doesn't take"
assert arg not in WRITE_TARGETS[tool], f"{tool}.{arg} is both a locator and a reference"
def test_the_fence_hook_skips_exactly_the_read_prefixes_the_server_names():
import re
from pathlib import Path
hook = Path(__file__).resolve().parents[1] / "plugin" / "hooks" / "grimoire_project_fence.sh"
found = re.search(r'^FENCE_READ_PREFIXES="([^"]*)"$', hook.read_text(), re.M)
assert found, "grimoire_project_fence.sh no longer declares FENCE_READ_PREFIXES"
assert tuple(found.group(1).split()) == write_scope.READ_PREFIXES
def test_no_write_tool_carries_a_read_prefix():
# The fence hook skips the server for these prefixes, so a write named
# like a read would pass unchecked.
from grimoire.mcp.server import _WRITE_TOOLS
read_named = sorted(t for t in _WRITE_TOOLS if t.startswith(write_scope.READ_PREFIXES))
assert not read_named, f"write tools named like reads: {read_named}"
def test_the_locator_scan_sees_the_arguments_it_guards():
# The guard above passes vacuously if the scan finds no parameters at all.
params = _write_tool_params()
assert "task_id" in params["update_task"] & LOCATOR_NAMES
assert "project_id" in params["create_task"] & LOCATOR_NAMES
@pytest.mark.parametrize("args, expected", [
({"project_id": 7, "kind": "issue"}, True),
({"project_id": 7, "kind": "issue", "status": "todo"}, True),
({"project_id": 7, "kind": "work"}, False),
({"project_id": 7}, False),
({"project_id": 7, "kind": "issue", "status": "in_progress"}, False),
({"project_id": 7, "kind": "issue", "milestone_id": 3}, False),
({"project_id": 7, "kind": "issue", "parent_id": 9}, False),
({"project_id": 7, "kind": "issue", "milestone_id": 0}, True),
])
def test_filing_an_issue_is_a_new_open_unplanned_issue(args, expected):
assert is_issue_filing("create_task", args) is expected
def test_only_create_task_files_an_issue():
assert is_issue_filing("update_task", {"task_id": 4, "kind": "issue"}) is False
assert is_issue_filing("create_records", {"project_id": 7, "kind": "issue"}) is False
async def test_ids_of_zero_or_below_locate_nothing():
# update_task's project_id=0 leaves it unchanged and -1 clears it.
assert await write_scope.tool_targets(1, "update_task", {"project_id": -1}) == set()
assert await write_scope.tool_targets(1, "create_task", {"project_id": 0}) == set()
assert await write_scope.tool_targets(1, "create_note", {"project_id": 12}) == {12}
async def test_reads_and_unknown_tools_land_nowhere():
assert await write_scope.tool_targets(1, "get_task", {"task_id": 5}) == set()
assert await write_scope.tool_targets(1, "no_such_tool", {"project_id": 5}) == set()
def _patched(landing: set[int], readable: bool = True):
return (
patch.object(write_scope, "tool_targets", AsyncMock(return_value=landing)),
patch("grimoire.services.access.can_read_project", AsyncMock(return_value=readable)),
patch.object(write_scope, "_titles", AsyncMock(return_value={2: "Home", 9: "Elsewhere"})),
)
async def test_a_write_in_the_sessions_own_project_goes_ahead():
a, b, c = _patched({2})
with a, b, c:
out = await check_act(1, 2, tool="update_task", args={"task_id": 5})
assert out == {"allowed": True, "targets": [], "issue": False, "reason": ""}
async def test_a_write_in_another_project_asks_and_says_how_to_file_an_issue_instead():
a, b, c = _patched({9})
with a, b, c:
out = await check_act(1, 2, tool="update_task", args={"task_id": 5})
assert out["allowed"] is False
assert out["targets"] == [{"id": 9, "title": "Elsewhere"}]
assert "Home (project 2)" in out["reason"]
assert "Elsewhere (project 9)" in out["reason"]
assert 'create_task(project_id=9, kind="issue")' in out["reason"]
async def test_filing_an_issue_in_another_project_goes_ahead():
a, b, c = _patched({9})
with a, b, c:
out = await check_act(1, 2, tool="create_task", args={"project_id": 9, "kind": "issue"})
assert out["allowed"] is True and out["issue"] is True
async def test_a_session_with_no_project_is_held_to_nothing():
a, b, c = _patched({9})
with a, b, c:
out = await check_act(1, 0, tool="update_task", args={"task_id": 5})
assert out["allowed"] is True
async def test_a_project_the_caller_cannot_read_is_not_named():
a, b, c = _patched({9}, readable=False)
with a, b, c:
out = await check_act(1, 2, tool="update_task", args={"task_id": 5})
assert out == {"allowed": True, "targets": [], "issue": False, "reason": ""}
async def test_a_directory_in_another_project_asks_in_the_words_of_the_act():
with patch("grimoire.services.access.can_read_project", AsyncMock(return_value=True)), \
patch.object(write_scope, "_titles", AsyncMock(return_value={2: "Home", 9: "Elsewhere"})):
out = await check_act(1, 2, target_project=9, act="git push in")
same = await check_act(1, 2, target_project=2, act="git push in")
assert out["allowed"] is False
assert "git push in Elsewhere (project 9)" in out["reason"]
assert same["allowed"] is True
async def test_the_route_reads_the_session_the_tool_and_its_input():
from grimoire.app import create_app
from grimoire.routes import plugin
app = create_app()
answer = {"allowed": False, "targets": [{"id": 9, "title": "x"}], "issue": False, "reason": "r"}
check = AsyncMock(return_value=dict(answer))
url = ("/api/plugin/write-scope?project_id=2"
"&tool=mcp__plugin_grimoire_grimoire__update_task"
"&input=%7B%22task_id%22%3A5%7D&act=x")
async with app.test_request_context(url):
g.user = SimpleNamespace(id=1)
with patch.object(plugin.write_scope_svc, "check_act", check), \
patch.object(plugin, "can_read_project", AsyncMock(return_value=True)):
resp = await plugin.write_scope.__wrapped__()
body = await resp.get_json()
assert body["allowed"] is False and body["session_project"] == 2
_, kwargs = check.call_args
assert check.call_args.args == (1, 2)
assert kwargs["tool"] == "update_task" and kwargs["args"] == {"task_id": 5}
async def test_the_route_resolves_a_target_repo_to_its_project():
from grimoire.app import create_app
from grimoire.routes import plugin
app = create_app()
check = AsyncMock(return_value={"allowed": True, "targets": [], "issue": False, "reason": ""})
url = "/api/plugin/write-scope?project_id=2&target_repo=https://forge.example/o/r.git&act=git%20push%20in"
async with app.test_request_context(url):
g.user = SimpleNamespace(id=1)
with patch.object(plugin.write_scope_svc, "check_act", check), \
patch.object(plugin, "can_read_project", AsyncMock(return_value=True)), \
patch.object(plugin.repo_bindings_svc, "resolve_project", AsyncMock(return_value=9)):
await plugin.write_scope.__wrapped__()
assert check.call_args.kwargs["target_project"] == 9
assert check.call_args.kwargs["act"] == "git push in"
async def test_a_session_whose_directory_resolves_to_nothing_belongs_to_the_project_it_entered():
from grimoire.app import create_app
from grimoire.routes import plugin
app = create_app()
check = AsyncMock(return_value={"allowed": True, "targets": [], "issue": False, "reason": ""})
url = ("/api/plugin/write-scope?fallback_project_id=7"
"&tool=add_task_log&input=%7B%22task_id%22%3A5%7D")
async with app.test_request_context(url):
g.user = SimpleNamespace(id=1)
with patch.object(plugin.write_scope_svc, "check_act", check), \
patch.object(plugin, "can_read_project", AsyncMock(return_value=True)):
await plugin.write_scope.__wrapped__()
assert check.call_args.args == (1, 7)
async def test_the_route_refuses_input_that_is_not_an_object():
from grimoire.app import create_app
from grimoire.routes import plugin
app = create_app()
async with app.test_request_context("/api/plugin/write-scope?tool=update_task&input=%5B1%5D"):
g.user = SimpleNamespace(id=1)
with patch.object(plugin, "can_read_project", AsyncMock(return_value=True)):
resp, status = await plugin.write_scope.__wrapped__()
assert status == 400