ci(plugin): add shellcheck + jq, and the fail-open smoke test they unlock
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Failing after 7s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / integration (push) Successful in 39s
CI & Build / Python tests (push) Successful in 47s
CI & Build / Build & push image (push) Successful in 21s

Per-job installs, not an image change. CI-runner's docs/process.md decision
checkpoint is explicit: "If only one project needs the dep, prefer that
project installing it per-job in their workflow — at least until a second
consumer arrives." Scribe is the only consumer, and shellcheck is not a
natural extension of a Python image's purpose. Promotion into ci-python is
filed as an issue on CI-runner rather than assumed here — same doc, step 1:
the maintainer's call goes in the issue, then the PR.

This also corrects something I got wrong earlier in this work: I cited rule
#5 as blocking a per-job install. Rule #5 is about language TOOLCHAINS via
setup-* actions, not small lint utilities, and CI-runner's own process doc
positively recommends per-job installs in exactly this case.

jq is load-bearing rather than convenient. Every hook opens with
`command -v jq || exit 0`, so without it a "runs and stays silent" smoke
test passes while exercising nothing — a green tick proving less than no
test at all. That is why the smoke test didn't ship with the first cut.

The smoke test pins the fail-open contract: each hook, with no credentials
and then against a refused connection, must exit 0. Three must also stay
silent; scribe_session_context.sh must NOT, because its static behavioural
floor is meant to survive having no credentials and no network — asserting
silence there would encode the opposite of the design.

Verified it can actually fail, rather than assuming: injected a non-zero
exit and separately a stray stdout write, and confirmed each is caught.

shellcheck and jq are both optional at runtime — missing either SKIPs its
check loudly rather than passing. A check that quietly no-ops is the exact
failure mode this file exists to prevent.

ci-requirements.md updated: jq + shellcheck recorded under per-job installs
(the input CI-runner's maintainer uses for the next promotion decision),
plus two stale entries corrected — the sheet claimed four jobs when there
are six, and listed `uv` as a per-job install when it has been in the image
since the ci-python Dockerfile started pip-installing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-28 22:39:55 -04:00
parent 1feef179d2
commit dd878bc498
3 changed files with 157 additions and 17 deletions
+130 -7
View File
@@ -17,13 +17,18 @@ The rule for the second one was already written down and was still missed. A
written rule that depends on being remembered during a long session is not a
control; this is.
WHAT IT DOESN'T COVER. shellcheck would be the right tool for the pattern
checks below and is NOT in `ci-python` (verified against CI-runner's Dockerfile
+ scripts/install-common.sh, not from memory — rule #37). Neither is `jq`, which
every hook requires and bails without, so a "runs and stays silent" smoke test
would pass vacuously today and prove nothing. Both need the image to grow those
two packages; until then this checks the specific bug classes that actually bit,
which is narrower than shellcheck but not nothing.
shellcheck and jq are NOT in `ci-python` (verified against CI-runner's Dockerfile
and scripts/install-common.sh, not from memory — rule #37). CI installs both
per-job, which is what CI-runner's own docs/process.md prescribes for a dep with
a single consumer: "If only one project needs the dep, prefer that project
installing it per-job in their workflow — at least until a second consumer
arrives." Promotion into the image is filed as an issue there rather than
assumed here.
Both are optional at runtime: without shellcheck the lint step is SKIPPED and
says so, and without jq the smoke test is skipped. A skipped check announces
itself loudly, because a check that quietly no-ops is the failure mode this
whole file exists to prevent.
Usage:
python3 scripts/check_plugin.py # all checks
@@ -33,7 +38,9 @@ from __future__ import annotations
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
@@ -59,6 +66,12 @@ def ok(msg: str) -> None:
print(f"ok {msg}")
def skip(msg: str) -> None:
# Loud on purpose. A check that quietly does nothing is indistinguishable
# from a check that passed — the exact confusion that let #2198 survive.
print(f"SKIP {msg}")
def hook_scripts() -> list[Path]:
return sorted(HOOKS_DIR.glob("*.sh"))
@@ -132,6 +145,114 @@ def check_patterns() -> None:
# --- the version bump ------------------------------------------------------
# --- shellcheck ------------------------------------------------------------
def check_shellcheck() -> None:
"""Real shell linting, where available.
The hand-rolled patterns above only know the bugs that already happened.
This is what catches the next one.
"""
exe = shutil.which("shellcheck")
if not exe:
skip("shellcheck not installed — install it to lint the hooks properly")
return
for script in hook_scripts():
proc = subprocess.run(
[exe, "--severity=warning", "--shell=bash", str(script)],
capture_output=True, text=True,
)
rel = script.relative_to(ROOT)
if proc.returncode != 0:
fail(f"{rel}: shellcheck\n{proc.stdout.strip()}")
else:
ok(f"{rel}: shellcheck")
# --- the fail-open contract ------------------------------------------------
# Every hook promises never to break the operator's session: unconfigured or
# unreachable, it exits 0. Three of them additionally promise SILENCE, because
# they are pure enrichment. scribe_session_context.sh is the exception by
# design — it always emits a static behavioural floor that needs no credentials
# and no network, so "silent" would be the wrong assertion for it.
#
# This is the contract that made #2198 invisible for weeks, so it is worth
# pinning: the bug and the healthy no-results case look identical from outside.
# Pinning it does NOT make the failure visible; it makes sure the fail-open
# behaviour is deliberate rather than accidental.
SMOKE_EVENTS: dict[str, str] = {
"scribe_autoinject.sh": json.dumps(
{"session_id": "smoke", "cwd": ".", "prompt": "a multi-line\nprompt\nhere"}
),
"scribe_prior_art.sh": json.dumps(
{"session_id": "smoke", "cwd": ".", "tool_name": "Edit",
"tool_input": {"file_path": "src/x.py", "new_string": "def f():\n pass\n"}}
),
"scribe_sync_processes.sh": json.dumps({"source": "startup"}),
"scribe_session_context.sh": json.dumps({"source": "startup"}),
}
# The one hook that legitimately produces output with no credentials.
STATIC_FLOOR = "scribe_session_context.sh"
def _run_hook(script: Path, event: str, env_extra: dict[str, str]) -> subprocess.CompletedProcess:
env = {k: v for k, v in os.environ.items()
if not k.startswith(("SCRIBE_", "CLAUDE_PLUGIN_OPTION_"))}
env.update(env_extra)
return subprocess.run(
["bash", str(script)], input=event, capture_output=True, text=True,
env=env, timeout=30,
)
def check_fail_open() -> None:
if not shutil.which("jq"):
# Without jq every hook bails at its first line, so this would pass
# while exercising nothing. Say so rather than bank a green tick.
skip("jq not installed — the hooks would exit at line 1, so this "
"check would pass without testing anything")
return
scenarios = [
("unconfigured", {}),
# Connection refused immediately — exercises the unreachable-instance
# path without waiting on a real network timeout.
("unreachable", {"SCRIBE_URL": "http://127.0.0.1:1", "SCRIBE_TOKEN": "x"}),
]
for script in hook_scripts():
rel = script.relative_to(ROOT)
event = SMOKE_EVENTS.get(script.name)
if event is None:
skip(f"{rel}: no smoke event defined")
continue
for label, env_extra in scenarios:
try:
proc = _run_hook(script, event, env_extra)
except subprocess.TimeoutExpired:
fail(f"{rel} [{label}]: hung — a hook must never block a session")
continue
if proc.returncode != 0:
fail(f"{rel} [{label}]: exited {proc.returncode}, must be 0 — "
f"a recall aid may never fail the operator's action")
continue
out = proc.stdout.strip()
if script.name == STATIC_FLOOR:
# Emits its bundled static tier regardless; that floor is the
# whole point of the two-tier design.
if not out:
fail(f"{rel} [{label}]: emitted nothing — the static "
f"behavioural floor must survive having no credentials")
else:
ok(f"{rel} [{label}]: exit 0, static floor present")
elif out:
fail(f"{rel} [{label}]: emitted output with no working instance:\n"
f" {out[:200]}")
else:
ok(f"{rel} [{label}]: exit 0, silent")
def _git(*args: str) -> tuple[int, str]:
proc = subprocess.run(
["git", *args], capture_output=True, text=True, cwd=ROOT
@@ -221,6 +342,8 @@ def main() -> int:
check_syntax()
check_patterns()
check_shellcheck()
check_fail_open()
if not args.no_version:
check_version_bump(args.base)