bvandeusen/FabledScribe
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases 53 Wiki Activity

feat(rules): the staleness sweep — which standing rules assert a fact nobody has confirmed (#3097, milestone 312 step 3)

Browse Source 
The query the last two steps were storage for. `rules_due_for_verification`
returns every rule carrying a `verify_with`, ordered by `verified_at` ASC
NULLS FIRST, each row carrying the check IN FULL — the opposite call from
rule_brief, because the reader is about to go and run it.

NULLS FIRST is the ordering this turns on. Postgres sorts NULLs last on an
ASC ordering, which would put the rules nobody has ever confirmed BEHIND
every rule someone once looked at. Exactly backwards: a claim with no
evidence at all outranks an old one.

Rules with no check never appear, and that is the property that keeps the
list worth reading. Most rules are decisions — no truth value, nothing to go
and check. If they appeared here the sweep would be the rulebook.

`mark_rule_verified(rule_id, still_true)` closes the loop, asymmetrically:
passing writes a stamp, FAILING WRITES NOTHING. There is no "verified false"
state because a rule whose check failed is not in a special condition, it is
wrong — and recording the failure as a flag would let it sit there being
false with the sweep satisfied that someone had looked. So it stays at the
top until someone corrects or retires it, and the response says so.

An unrecognised `tier` filter raises rather than falling back. _valid_tier's
silent always_on default is right for a WRITE — a typo should leave a rule
binding — and wrong for a FILTER, where the same fallback quietly answers a
different question and returns a short list that reads as good news.

Deliberately NOT filterable by project: a project reaches rules through
project scope, subscriptions, always-on rulebooks and exclusions, and a
filter missing one of those paths would UNDER-report — the exact failure
this surface exists to prevent. Said so in the docstring rather than
shipping a half-correct filter.

Ownership-scoped like every other rule read (owned rulebook, or owned
project), in ONE statement with an OR across the XOR rather than two queries
merged in Python, so the ordering is the database's and cannot disagree with
itself. Note that rules have no sharing ACL in this schema — no rule_shares,
no rulebook_shares — so there is no wider set for access.py to consult here.

Also fixes a test title that had been lying for ten tools: "all sixteen
tools" asserted 26. The number now lives only in the assertion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-08-27 13:43:57 -04:00
committed by bvandeusen
co-authored by Claude Opus 5
parent 874f7cacdb
commit b97f57ee7f
7 changed files with 560 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/scribe/mcp/tools/rulebooks.py
+92
View File
@@ -691,6 +691,97 @@ async def unrelate_rules(relation_id: int) -> dict:
raise ValueError(f"relation {relation_id} not found")
return {"deleted": relation_id}
# ── The staleness sweep (milestone 312) ────────────────────────────────
async def rules_due_for_verification(
older_than_days: int = 0, tier: str = "", never_only: bool = False,
) -> dict:
"""Which standing rules assert a FACT that nobody has confirmed lately.
A rulebook holds two kinds of thing. Most rules are DECISIONS — how the
operator wants to work. They have no truth value and cannot rot. A few
assert a fact about someone else's software: what a CI runner does, which
tools exist, what a setting is currently set to. Those go false silently,
with nobody present, and they keep being handed to every session as
binding instructions long after they stopped being true.
This lists the second kind, oldest verification first, never-checked at
the top. Each row carries the rule's `verify_with` in full — you are
about to go and run it — plus `expires_when`, and `days_since_verified`.
Reach for it when you are curating the rulebook, when a rule's advice
just contradicted what you observed, or periodically. Then, for each row:
run the check, and call mark_rule_verified with what you found.
Rules with no `verify_with` never appear here. That is correct: they are
decisions, and there is nothing to go and check. Do not "fix" their
absence by giving them checks — the list is only worth reading while
everything on it genuinely can go false.
Args:
older_than_days: only rules last verified longer ago than this.
Never-checked rules always qualify. 0 = no age filter.
tier: "always_on" or "conditional" to narrow. An always-on constraint
that has gone false is the expensive kind — it is preloaded into
every session, so a wrong one is wrong everywhere at once.
never_only: only rules nobody has ever verified.
NOT filterable by project, deliberately: a project reaches rules through
project scope, subscriptions, always-on rulebooks and exclusions, and a
filter that missed one of those paths would UNDER-report — which is the
exact failure this whole surface exists to prevent. Read the whole list.
"""
uid = current_user_id()
rules = await rulebooks_svc.rules_due_for_verification(
uid, older_than_days=older_than_days, tier=tier, never_only=never_only,
)
return {
"rules": [rulebooks_svc.verification_row(r) for r in rules],
"total": len(rules),
}
async def mark_rule_verified(rule_id: int, still_true: bool = True) -> dict:
"""Record that you ran a rule's check — and what it said.
Call this AFTER actually running the rule's `verify_with`, never on the
strength of the rule sounding plausible. A stamp nobody earned is worse
than no stamp: it moves the rule to the bottom of the sweep and buys it
another long silence.
`still_true=False` writes NOTHING. A rule whose check failed is not in a
special state to be recorded — it is WRONG, and the only honest next
moves are to correct it, retire it, or find out why. So it stays at the
top of the sweep until someone deals with it, and the response tells you
what the rule said would end it.
Args:
rule_id: the rule whose check you ran.
still_true: True if the check passed. False if the fact it asserts is
no longer true — say so, that is the outcome worth having.
"""
uid = current_user_id()
rule = await rulebooks_svc.mark_rule_verified(rule_id, uid, still_true)
if rule is None:
raise ValueError(
f"rule {rule_id} not found, or carries no verify_with "
f"(nothing to verify is not the same as verified)"
)
data = await rulebooks_svc.rule_detail(uid, rule)
if still_true:
data["verified"] = True
return data
data["verified"] = False
data["next"] = (
"This rule is no longer true and is still binding on every session "
"that loads it. Correct it with update_rule, retire it with "
"delete_rule, or open a task to work out what replaced it. Its "
"verified_at is deliberately untouched, so it stays at the top of "
"rules_due_for_verification until one of those happens."
)
return data
def register(mcp) -> None:
for fn in (
list_rulebooks, get_rulebook, create_rulebook, update_rulebook, delete_rulebook,
@@ -702,5 +793,6 @@ def register(mcp) -> None:
suppress_rule_for_project, unsuppress_rule_for_project,
suppress_topic_for_project, unsuppress_topic_for_project,
exclude_always_on_rulebook, include_always_on_rulebook,
rules_due_for_verification, mark_rule_verified,
):
mcp.tool(name=fn.__name__)(fn)
src/scribe/routes/rulebooks.py
+53
View File
@@ -390,3 +390,56 @@ async def create_project_rule(project_id: int):
return jsonify(await rulebooks_svc.rule_detail(
get_current_user_id(), rule, data.get("system_ids"),
)), 201
# ── The staleness sweep (milestone 312) ────────────────────────────────
@rulebooks_bp.get("/rules-due-for-verification")
@login_required
async def rules_due_for_verification():
"""Rules that carry a check, oldest verification first, never-checked top.
Query params: older_than_days, tier, never_only. A rule with no
`verify_with` never appears — it is a decision, not a fact.
"""
uid = get_current_user_id()
args = request.args
try:
older = int(args.get("older_than_days", 0) or 0)
except ValueError:
return jsonify({"error": "older_than_days must be an integer"}), 400
try:
rules = await rulebooks_svc.rules_due_for_verification(
uid,
older_than_days=older,
tier=args.get("tier", ""),
never_only=args.get("never_only", "").lower() in ("1", "true", "yes"),
)
except ValueError as exc:
# An unrecognised tier is a 400, not a silently narrowed result set:
# a filter that quietly answers a different question is the failure
# this whole surface exists to catch.
return jsonify({"error": str(exc)}), 400
return jsonify({
"rules": [rulebooks_svc.verification_row(r) for r in rules],
"total": len(rules),
})
@rulebooks_bp.post("/rules/<int:rule_id>/verify")
@login_required
async def mark_rule_verified(rule_id: int):
"""Record that the rule's check was run. Body: {"still_true": bool}.
`still_true: false` writes nothing — a rule whose check failed is wrong,
not in a recordable state — so it stays at the top of the sweep.
"""
data = await request.get_json() or {}
uid = get_current_user_id()
still_true = data.get("still_true", True)
rule = await rulebooks_svc.mark_rule_verified(rule_id, uid, bool(still_true))
if rule is None:
return jsonify({"error": "rule not found, or carries no verify_with"}), 404
payload = await rulebooks_svc.rule_detail(uid, rule)
payload["verified"] = bool(still_true)
return jsonify(payload)
src/scribe/services/rulebooks.py
+148 -1
View File
@@ -11,7 +11,7 @@ import logging
from collections.abc import Iterable
from typing import Optional
from sqlalchemy import delete as sql_delete, insert, or_, select
from sqlalchemy import and_, delete as sql_delete, insert, or_, select
from scribe.models import async_session
from scribe.models.system import System
@@ -1361,3 +1361,150 @@ def rules_payload(applicable: dict) -> dict:
"suppressed_topics": applicable.get("suppressed_topics", []),
"excluded_always_on": applicable.get("excluded_always_on", []),
}
# ── The staleness sweep (milestone 312) ────────────────────────────────
async def rules_due_for_verification(
user_id: int,
older_than_days: int = 0,
tier: str = "",
never_only: bool = False,
) -> list[Rule]:
"""Rules that carry a check, oldest verification first, never-checked top.
THE QUERY THIS MILESTONE EXISTS FOR. `verify_with` and `expires_when` are
storage; this is what turns them into something that gets acted on. The
307 audit cost a session and found four broken rules by luck — this makes
the same question a list, and staleness measurable by age instead of
discoverable by accident.
Ordered `verified_at` ASC NULLS FIRST: never-checked outranks
checked-long-ago, because a rule nobody has ever confirmed is a claim
with no evidence behind it at all.
Rules with no `verify_with` never appear. That is not an omission — they
are decisions, there is nothing to go and check, and listing them would
dilute the result until nobody reads it.
Ownership-scoped exactly like list_rules: a rule reached through an owned
rulebook, or scoped to an owned project. Rules have no sharing ACL in this
schema — no rule_shares, no rulebook_shares — so there is no wider set to
consult here, unlike notes and projects.
Args:
user_id: whose rules.
older_than_days: only rules last verified longer ago than this.
Never-checked rules always qualify — they are the most overdue
thing there is. 0 = no age filter.
tier: "always_on" or "conditional" to narrow. Raises on anything else
rather than falling back: _valid_tier's silent always_on default
is right for a WRITE (the safe direction is to keep binding), and
wrong for a FILTER, where it would quietly answer a different
question than the one asked.
never_only: only rules that have never been verified.
"""
from datetime import datetime, timedelta, timezone
from scribe.models.project import Project
if tier and tier not in TIERS:
raise ValueError(f"tier must be one of {TIERS}, got {tier!r}")
async with async_session() as session:
stmt = (
select(Rule)
.outerjoin(RulebookTopic, Rule.topic_id == RulebookTopic.id)
.outerjoin(Rulebook, RulebookTopic.rulebook_id == Rulebook.id)
.outerjoin(Project, Rule.project_id == Project.id)
.where(
Rule.deleted_at.is_(None),
Rule.verify_with.is_not(None),
# One statement rather than two queries merged in Python, so
# the ordering below is the database's and cannot disagree
# with itself across the two halves of the XOR.
or_(
and_(
Rulebook.owner_user_id == user_id,
Rulebook.deleted_at.is_(None),
RulebookTopic.deleted_at.is_(None),
),
Project.user_id == user_id,
),
)
)
if tier:
stmt = stmt.where(Rule.tier == tier)
if never_only:
stmt = stmt.where(Rule.verified_at.is_(None))
elif older_than_days > 0:
cutoff = datetime.now(timezone.utc) - timedelta(days=older_than_days)
stmt = stmt.where(
or_(Rule.verified_at.is_(None), Rule.verified_at < cutoff)
)
stmt = stmt.order_by(Rule.verified_at.asc().nullsfirst(), Rule.id)
return list((await session.execute(stmt)).scalars().all())
def verification_row(rule: Rule) -> dict:
"""One row of the sweep — the CHECK in full, unlike rule_brief.
The opposite call from a listing: here the caller is about to go and run
the check, so the text they need is the point of the payload rather than
the bloat. `days_since` is computed rather than left to the reader,
because "2026-06-14" and "74 days" prompt different reactions and only
one of them is the question being asked.
"""
from datetime import datetime, timezone
days = None
if rule.verified_at is not None:
stamp = rule.verified_at
if stamp.tzinfo is None:
stamp = stamp.replace(tzinfo=timezone.utc)
days = (datetime.now(timezone.utc) - stamp).days
return {
"id": rule.id,
"title": rule.title,
"statement": rule.statement,
"tier": rule.tier,
"topic_id": rule.topic_id,
"project_id": rule.project_id,
"when_to_apply": rule.when_to_apply or "",
"verify_with": rule.verify_with or "",
"expires_when": rule.expires_when or "",
"last_verified": last_verified_label(rule),
"days_since_verified": days,
}
async def mark_rule_verified(
rule_id: int, user_id: int, still_true: bool = True,
) -> Optional[Rule]:
"""Stamp a rule as verified — or, when the check FAILED, refuse to.
A failing check is the outcome worth having, and the asymmetry is
deliberate: passing writes a stamp, failing writes nothing. There is no
"verified false" state to record, because a rule whose check failed is
not a rule in a special condition — it is a rule that is WRONG, and the
only honest resolutions are to correct it, retire it, or find out why.
Recording the failure as a flag would let it sit there being false with
the sweep quietly satisfied that someone had looked.
So a failed check leaves `verified_at` untouched, and the rule stays at
the top of the sweep until someone actually deals with it.
Returns None when the rule is not found, not owned, or carries no
`verify_with` — nothing to verify is a different answer from verified.
"""
from datetime import datetime, timezone
async with async_session() as session:
rule = await _fetch_owned_rule(session, rule_id, user_id)
if rule is None or not rule.verify_with:
return None
if still_true:
rule.verified_at = datetime.now(timezone.utc)
await session.commit()
await session.refresh(rule)
return rule