ci(plugin): add shellcheck + jq, and the fail-open smoke test they unlock
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Failing after 7s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / integration (push) Successful in 39s
CI & Build / Python tests (push) Successful in 47s
CI & Build / Build & push image (push) Successful in 21s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Failing after 7s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / integration (push) Successful in 39s
CI & Build / Python tests (push) Successful in 47s
CI & Build / Build & push image (push) Successful in 21s
Per-job installs, not an image change. CI-runner's docs/process.md decision checkpoint is explicit: "If only one project needs the dep, prefer that project installing it per-job in their workflow — at least until a second consumer arrives." Scribe is the only consumer, and shellcheck is not a natural extension of a Python image's purpose. Promotion into ci-python is filed as an issue on CI-runner rather than assumed here — same doc, step 1: the maintainer's call goes in the issue, then the PR. This also corrects something I got wrong earlier in this work: I cited rule #5 as blocking a per-job install. Rule #5 is about language TOOLCHAINS via setup-* actions, not small lint utilities, and CI-runner's own process doc positively recommends per-job installs in exactly this case. jq is load-bearing rather than convenient. Every hook opens with `command -v jq || exit 0`, so without it a "runs and stays silent" smoke test passes while exercising nothing — a green tick proving less than no test at all. That is why the smoke test didn't ship with the first cut. The smoke test pins the fail-open contract: each hook, with no credentials and then against a refused connection, must exit 0. Three must also stay silent; scribe_session_context.sh must NOT, because its static behavioural floor is meant to survive having no credentials and no network — asserting silence there would encode the opposite of the design. Verified it can actually fail, rather than assuming: injected a non-zero exit and separately a stray stdout write, and confirmed each is caught. shellcheck and jq are both optional at runtime — missing either SKIPs its check loudly rather than passing. A check that quietly no-ops is the exact failure mode this file exists to prevent. ci-requirements.md updated: jq + shellcheck recorded under per-job installs (the input CI-runner's maintainer uses for the next promotion decision), plus two stale entries corrected — the sheet claimed four jobs when there are six, and listed `uv` as a per-job install when it has been in the image since the ci-python Dockerfile started pip-installing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
+130
-7
@@ -17,13 +17,18 @@ The rule for the second one was already written down and was still missed. A
|
||||
written rule that depends on being remembered during a long session is not a
|
||||
control; this is.
|
||||
|
||||
WHAT IT DOESN'T COVER. shellcheck would be the right tool for the pattern
|
||||
checks below and is NOT in `ci-python` (verified against CI-runner's Dockerfile
|
||||
+ scripts/install-common.sh, not from memory — rule #37). Neither is `jq`, which
|
||||
every hook requires and bails without, so a "runs and stays silent" smoke test
|
||||
would pass vacuously today and prove nothing. Both need the image to grow those
|
||||
two packages; until then this checks the specific bug classes that actually bit,
|
||||
which is narrower than shellcheck but not nothing.
|
||||
shellcheck and jq are NOT in `ci-python` (verified against CI-runner's Dockerfile
|
||||
and scripts/install-common.sh, not from memory — rule #37). CI installs both
|
||||
per-job, which is what CI-runner's own docs/process.md prescribes for a dep with
|
||||
a single consumer: "If only one project needs the dep, prefer that project
|
||||
installing it per-job in their workflow — at least until a second consumer
|
||||
arrives." Promotion into the image is filed as an issue there rather than
|
||||
assumed here.
|
||||
|
||||
Both are optional at runtime: without shellcheck the lint step is SKIPPED and
|
||||
says so, and without jq the smoke test is skipped. A skipped check announces
|
||||
itself loudly, because a check that quietly no-ops is the failure mode this
|
||||
whole file exists to prevent.
|
||||
|
||||
Usage:
|
||||
python3 scripts/check_plugin.py # all checks
|
||||
@@ -33,7 +38,9 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -59,6 +66,12 @@ def ok(msg: str) -> None:
|
||||
print(f"ok {msg}")
|
||||
|
||||
|
||||
def skip(msg: str) -> None:
|
||||
# Loud on purpose. A check that quietly does nothing is indistinguishable
|
||||
# from a check that passed — the exact confusion that let #2198 survive.
|
||||
print(f"SKIP {msg}")
|
||||
|
||||
|
||||
def hook_scripts() -> list[Path]:
|
||||
return sorted(HOOKS_DIR.glob("*.sh"))
|
||||
|
||||
@@ -132,6 +145,114 @@ def check_patterns() -> None:
|
||||
|
||||
# --- the version bump ------------------------------------------------------
|
||||
|
||||
# --- shellcheck ------------------------------------------------------------
|
||||
|
||||
def check_shellcheck() -> None:
|
||||
"""Real shell linting, where available.
|
||||
|
||||
The hand-rolled patterns above only know the bugs that already happened.
|
||||
This is what catches the next one.
|
||||
"""
|
||||
exe = shutil.which("shellcheck")
|
||||
if not exe:
|
||||
skip("shellcheck not installed — install it to lint the hooks properly")
|
||||
return
|
||||
for script in hook_scripts():
|
||||
proc = subprocess.run(
|
||||
[exe, "--severity=warning", "--shell=bash", str(script)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
rel = script.relative_to(ROOT)
|
||||
if proc.returncode != 0:
|
||||
fail(f"{rel}: shellcheck\n{proc.stdout.strip()}")
|
||||
else:
|
||||
ok(f"{rel}: shellcheck")
|
||||
|
||||
|
||||
# --- the fail-open contract ------------------------------------------------
|
||||
|
||||
# Every hook promises never to break the operator's session: unconfigured or
|
||||
# unreachable, it exits 0. Three of them additionally promise SILENCE, because
|
||||
# they are pure enrichment. scribe_session_context.sh is the exception by
|
||||
# design — it always emits a static behavioural floor that needs no credentials
|
||||
# and no network, so "silent" would be the wrong assertion for it.
|
||||
#
|
||||
# This is the contract that made #2198 invisible for weeks, so it is worth
|
||||
# pinning: the bug and the healthy no-results case look identical from outside.
|
||||
# Pinning it does NOT make the failure visible; it makes sure the fail-open
|
||||
# behaviour is deliberate rather than accidental.
|
||||
SMOKE_EVENTS: dict[str, str] = {
|
||||
"scribe_autoinject.sh": json.dumps(
|
||||
{"session_id": "smoke", "cwd": ".", "prompt": "a multi-line\nprompt\nhere"}
|
||||
),
|
||||
"scribe_prior_art.sh": json.dumps(
|
||||
{"session_id": "smoke", "cwd": ".", "tool_name": "Edit",
|
||||
"tool_input": {"file_path": "src/x.py", "new_string": "def f():\n pass\n"}}
|
||||
),
|
||||
"scribe_sync_processes.sh": json.dumps({"source": "startup"}),
|
||||
"scribe_session_context.sh": json.dumps({"source": "startup"}),
|
||||
}
|
||||
|
||||
# The one hook that legitimately produces output with no credentials.
|
||||
STATIC_FLOOR = "scribe_session_context.sh"
|
||||
|
||||
|
||||
def _run_hook(script: Path, event: str, env_extra: dict[str, str]) -> subprocess.CompletedProcess:
|
||||
env = {k: v for k, v in os.environ.items()
|
||||
if not k.startswith(("SCRIBE_", "CLAUDE_PLUGIN_OPTION_"))}
|
||||
env.update(env_extra)
|
||||
return subprocess.run(
|
||||
["bash", str(script)], input=event, capture_output=True, text=True,
|
||||
env=env, timeout=30,
|
||||
)
|
||||
|
||||
|
||||
def check_fail_open() -> None:
|
||||
if not shutil.which("jq"):
|
||||
# Without jq every hook bails at its first line, so this would pass
|
||||
# while exercising nothing. Say so rather than bank a green tick.
|
||||
skip("jq not installed — the hooks would exit at line 1, so this "
|
||||
"check would pass without testing anything")
|
||||
return
|
||||
|
||||
scenarios = [
|
||||
("unconfigured", {}),
|
||||
# Connection refused immediately — exercises the unreachable-instance
|
||||
# path without waiting on a real network timeout.
|
||||
("unreachable", {"SCRIBE_URL": "http://127.0.0.1:1", "SCRIBE_TOKEN": "x"}),
|
||||
]
|
||||
for script in hook_scripts():
|
||||
rel = script.relative_to(ROOT)
|
||||
event = SMOKE_EVENTS.get(script.name)
|
||||
if event is None:
|
||||
skip(f"{rel}: no smoke event defined")
|
||||
continue
|
||||
for label, env_extra in scenarios:
|
||||
try:
|
||||
proc = _run_hook(script, event, env_extra)
|
||||
except subprocess.TimeoutExpired:
|
||||
fail(f"{rel} [{label}]: hung — a hook must never block a session")
|
||||
continue
|
||||
if proc.returncode != 0:
|
||||
fail(f"{rel} [{label}]: exited {proc.returncode}, must be 0 — "
|
||||
f"a recall aid may never fail the operator's action")
|
||||
continue
|
||||
out = proc.stdout.strip()
|
||||
if script.name == STATIC_FLOOR:
|
||||
# Emits its bundled static tier regardless; that floor is the
|
||||
# whole point of the two-tier design.
|
||||
if not out:
|
||||
fail(f"{rel} [{label}]: emitted nothing — the static "
|
||||
f"behavioural floor must survive having no credentials")
|
||||
else:
|
||||
ok(f"{rel} [{label}]: exit 0, static floor present")
|
||||
elif out:
|
||||
fail(f"{rel} [{label}]: emitted output with no working instance:\n"
|
||||
f" {out[:200]}")
|
||||
else:
|
||||
ok(f"{rel} [{label}]: exit 0, silent")
|
||||
|
||||
|
||||
def _git(*args: str) -> tuple[int, str]:
|
||||
proc = subprocess.run(
|
||||
["git", *args], capture_output=True, text=True, cwd=ROOT
|
||||
@@ -221,6 +342,8 @@ def main() -> int:
|
||||
|
||||
check_syntax()
|
||||
check_patterns()
|
||||
check_shellcheck()
|
||||
check_fail_open()
|
||||
if not args.no_version:
|
||||
check_version_bump(args.base)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user