scripts.check_spec_authority

Reference › scripts › check_spec_authority

The spec-authority gate (RUL-013): a @traces decorator on a test may cite only a requirement id that some spec defines. It fails when a test cites an id that no spec contains. tests/scripts/test_check_spec_authority.py imports it.

Invocation: main takes --roots (what to scan; defaults are built in), --specs-dir (the spec corpus, the ground truth), --max (a ratchet ceiling on the number of ungrounded ids; default 0, to be left at zero once known fabrications are resolved) and --json. Relative paths resolve against the repository root, which repo_root finds.

find_violations loads the deployed AST toolkit (load_ast_toolkit) to read decorators without importing test code, keeps only test symbols (is_test_symbol) and compares each cited id with broad_ground_truth, the set of ids defined anywhere in the specs directory. Rows are reported with display paths (_display_path) and the run fails when their count exceeds the ceiling.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

No internal imports and no importers.

API

Constants

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/scripts/check_spec_authority.py

#!/usr/bin/env python3
"""Spec-authority gate (RUL-013) — a requirement id cited by a TEST must be
grounded in the spec corpus.

RUL-013 has two halves. The workflow half ("an agent asked the user instead of
reading the spec") is unmechanizable and stays judgment. The EVIDENCE half is
deterministic: when someone resolves a contract from a test instead of from a
spec, they must invent a requirement id to cite, because no spec defines the
behaviour they are asserting. That fabricated id is a permanent, corpus-wide,
machine-checkable artifact — and until this gate, nothing looked for it in
``tests/``.

WHAT THIS GATE DOES
    Runs the AST toolkit's ``trace_lint`` over ``tests/`` (REUSED by path, not
    reimplemented — the same pattern as ``scripts/ast_report.py::_load_tool``
    and ``scripts/trace_intel.py::_load_parse_spec_requirements``) and keeps
    ONLY findings where::

        rule == "dangling"   and   the decorated symbol is a test
                                   (name starts with ``test_`` / ``Test``)

    Ratchets on ``--max``, mirroring ``scripts/check_req_format.py --max`` and
    ``scripts/pyright_ratchet.py`` (now in codetools:
    src/codetools/refactor/pyright/ratchet.py). Exit 1 when the count exceeds the ceiling.

WHAT IT DELIBERATELY DOES NOT FLAG (each verified against the live tree; each
would be a false accusation)
    * Fixture classes that exercise the ``@traces`` machinery itself
      (``tests/core/test_traceability.py::Bad``, ``tests/test_core.py::Foo`` and
      ``::ConcreteEntity``). Their ids are synthetic ON PURPOSE — flagging them
      would demand defanging the rejection tests. The ``test_``/``Test`` naming
      rule excludes all three structurally, with no hand-maintained allowlist,
      because a fixture class is never named ``test_*``.
    * ``spec-mismatch``. Those ids are REAL (e.g. the REQ-002 family lives in
      ``specs/001-kernel/checklists/requirements.md``); only the decorator's
      ``spec=`` label is stale. Including that rule would manufacture false
      accusations against correct tests citing correct requirements.
    * ``shotgun`` / ``malformed`` / ``placeholder`` — style and format concerns
      already owned by ``scripts/check_req_format.py`` and trace-lint's src lane.
    * ``src/orchestrata_ultimate`` — out of scope. This ruling's distinctive
      clause is "never from a TEST"; src has its own separate trace-lint
      backlog.
    * REQ ids in docstrings, comments and string literals — only real
      ``@traces(...)`` decorator nodes are read.

GROUND-TRUTH HARDENING (load-bearing)
    ``trace_lint`` grounds ids in ``specs/*/checklists/requirements.md`` only.
    A spec that documents its requirements as prose, or whose checklist has not
    landed yet, would make legitimate ids look dangling. This gate therefore
    unions in the BROAD corpus set — any ``REQ-`` token in any ``specs/**/*.md``
    — before reporting. Measured 2026-08-05: broad 5540 ids, narrow checklist
    5413, and the broad set is a strict superset. That makes false accusation
    structurally impossible: the gate can only ever UNDER-report a real
    fabrication, never invent one. An under-reporting gate is honest; an
    over-reporting one poisons the well and gets switched off.

Wired in ``scripts/health_check.py`` alongside the other corpus-scoped ruling
gates. Not a git hook: the check needs the whole spec ground truth plus the
whole ``tests/`` tree, so a per-staged-file pass is the wrong shape.

Usage::

    python scripts/check_spec_authority.py               # ceiling 0
    python scripts/check_spec_authority.py --max 8       # ratchet
    python scripts/check_spec_authority.py --json
"""

from __future__ import annotations

import argparse
import importlib.util
import json
import os
import re
import sys
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Any

# The ratified ruling this gate enforces, verbatim. ``scripts/rulings_lint.py``
# resolves a ``- gate <script> <rule-id>`` directive by requiring <rule-id> to
# be a literal substring of this file, so the full text lives here — and it
# doubles as the honest statement of what the gate does and does not claim.
RULING = """RUL-013 — "Specs are authoritative — resolve behaviour/contract decisions from specs/SPEC-000..105 + @traces(reqs=[...]), never from a test and never from a guess."

VERDICT: mechanizable, in one narrow but real half — and the violation it catches is sitting in the tree right now, undetected by every existing gate.

The ruling has two halves. The workflow half ("an agent asked the user instead of reading the spec") is genuinely unmechanizable, exactly as the page's Verification section already says. The EVIDENCE half is not: when someone resolves a contract from a test instead of from a spec, they leave a permanent artifact — a requirement id cited by a test that no spec defines. That artifact is deterministic, corpus-wide, and currently ungated.

MEASURED PROOF THE VIOLATION IS REAL AND LIVE:
specs/003-intent/checklists/requirements.md defines REQ-003-FR24-001 .. -004. Nothing else. Yet eight test functions across six files carry:
  @traces(spec="003", story="FR-024", reqs=["REQ-003-FR24-100"]) ... through "-106"
  tests/e2e/test_full_pipeline.py:85, tests/orchestrator/test_pipeline_e2e.py:{51,60,80},
  tests/orchestrator/test_pipeline_e2e_analysis.py:43, tests/orchestrator/test_pipeline_e2e_docs_session.py:{65,90},
  tests/orchestrator/test_pipeline_e2e_review.py:34
Seven requirement ids that exist in NO spec, anywhere (grep of specs/ src/ plans/ docs/ returns nothing). Someone needed a requirement to cite for the pipeline e2e tests, spec 003 did not define one, so they invented a plausible continuation of the family. That is RUL-013's failure mode verbatim: the contract was resolved from a test.

WHY NO EXISTING GATE SEES IT (I read all three before designing):
- scripts/check_req_format.py checks GRAMMAR only. REQ-003-FR24-107 matches its canonical `{SPEC}-{SEG}{IDX}-{SEQ}` production perfectly. It passes today at --max 0 (rc=0, verified) and is structurally blind to whether an id EXISTS.
- serialize_codebase_ast.py trace-lint HAS the right rule (`dangling`) but is scoped to src/orchestrata_ultimate only (AGENTS.md two-lane rule, echoed at code-review.py:1331), currently reports 120 errors there, and — this is the point — is invoked by NOTHING: no health_check entry, no git hook, no CI. It is a built-but-never-used capability, the same defect RUL-023 names.
- trace_intel.py untraced-reqs / unspecced run the INVERSE direction (spec ids with no code) and are advisory.
So tests/ is completely ungated for requirement groundedness. That is the RUL-013 gap, and it is the one lane where the ruling's "never from a test" clause actually bites.

DESIGN: scripts/check_spec_authority.py — a thin gate that REUSES trace_lint (loaded by path, the established scripts/ pattern from trace_intel._load_parse_spec_requirements and ast_report._load_tool), runs it over tests/, and keeps only findings where rule == "dangling" AND the decorated symbol name starts with test_/Test. Ratchet via --max, mirroring check_req_format --max and pyright_ratchet (now in codetools: src/codetools/refactor/pyright/ratchet.py).

One hardening beyond stock trace_lint: use the BROAD ground truth (any REQ- token in any specs/**/*.md), not just specs/*/checklists/requirements.md. Measured: broad = 5505 ids, narrow checklist = 5413. All seven fabricated ids are absent from BOTH, so the gap does not bite today — but a future prose-only spec would make legitimate ids look dangling. Widening the ground truth makes false accusation structurally impossible: the gate can only ever under-report, never manufacture a violation.

LANDING PLAN: wire at --max 8 (today's exact count) so it fires on any NEW fabrication from day one; fix the seven ids; pin --max 0.

DECISION FOR THE USER (surfacing, not deferring): the RUL-013-correct fix for the seven ids is to WRITE THE SPEC — add REQ-003-FR24-005..011 to specs/003-intent/checklists/requirements.md describing the pipeline e2e behaviours those tests genuinely verify — rather than re-pointing the tests at unrelated existing ids. The tests describe real behaviour spec 003 failed to write down; the ruling says resolve from the spec, so the spec is what should change. That edits the authoritative spec corpus, so it is your call, not mine.

COLLATERAL BROKEN WINDOW FOUND (separate from RUL-013, reporting per fix-on-sight): .pre-commit-config.yaml's `req-id-format` hook — and every other hook in that file — NEVER EXECUTES in this checkout. `git config core.hooksPath` = scripts/hooks, which shadows .git/hooks entirely, and scripts/hooks/pre-commit never calls `pre-commit run` (grep count: 0). docs/req-id-grammar.md:7 claims check_req_format is "enforced as pre-commit hook req-id-format, pinned --max 0". That claim is false. Nothing has drifted yet (manual run is green), but the enforcement is imaginary. Fix: add check_req_format.py --max 0 to health_check.py's gate list, or make scripts/hooks/pre-commit shell out to `pre-commit run`."""

# The AST toolkit is resolved from THIS file's directory, never from
# ORCHESTRATA_ROOT: the gate must keep using the real toolkit even when it is
# pointed at a throwaway corpus (which is exactly how its own tests drive it).
_SCRIPTS_DIR = Path(__file__).resolve().parent
_TOOL_PATH = _SCRIPTS_DIR / "serialize_codebase_ast.py"

DEFAULT_ROOTS: tuple[str, ...] = ("tests",)
DEFAULT_SPECS_DIR = "specs"

# Deliberately looser than trace-lint's checklist-shape ground truth: ANY REQ
# token, in ANY spec markdown. See "GROUND-TRUTH HARDENING" above.
_BROAD_REQ_RE = re.compile(r"\b(REQ-[A-Za-z0-9][A-Za-z0-9-]*)")

# The one rule of trace-lint that is RUL-013's subject. See the module
# docstring for why every other rule is excluded rather than merely unused.
DANGLING_RULE = "dangling"


def repo_root(env: Mapping[str, str] | None = None) -> Path:
    """Repo the gate is pointed at. ``ORCHESTRATA_ROOT`` wins (same seam as
    ``scripts/check_req_format.py``); otherwise the checkout this file is in."""
    environ = os.environ if env is None else env
    raw = environ.get("ORCHESTRATA_ROOT", "")
    return Path(raw).resolve() if raw else _SCRIPTS_DIR.parent


def load_ast_toolkit(tool_path: Path = _TOOL_PATH) -> Any:
    """Load ``scripts/serialize_codebase_ast.py`` by path — ``scripts/`` is not
    an importable package, so this is the established pattern (see
    ``scripts/ast_report.py::_load_tool``)."""
    spec = importlib.util.spec_from_file_location("serialize_codebase_ast", tool_path)
    if spec is None or spec.loader is None:
        raise ImportError(f"cannot load AST toolkit from {tool_path}")
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod


def broad_ground_truth(specs_dir: Path) -> set[str]:
    """Every REQ id mentioned anywhere in ``<specs_dir>/**/*.md``."""
    ids: set[str] = set()
    if not specs_dir.is_dir():
        return ids
    for path in sorted(specs_dir.rglob("*.md")):
        try:
            text = path.read_text(encoding="utf-8")
        except OSError, UnicodeDecodeError:
            continue
        ids.update(m.group(1) for m in _BROAD_REQ_RE.finditer(text))
    return ids


def is_test_symbol(name: str) -> bool:
    """A pytest-collected symbol. Excludes the fixture classes that exist to
    exercise the ``@traces`` decorator itself — those are never named
    ``test_*``, so no allowlist has to be maintained by hand."""
    return name.startswith("test_") or name.startswith("Test")


def find_violations(
    roots: Sequence[Path],
    specs_dir: Path,
    tool: Any | None = None,
) -> list[dict[str, Any]]:
    """Requirement ids cited by a test decorator and defined by no spec."""
    present = [str(r) for r in roots if r.exists()]
    if not present:
        return []
    toolkit = load_ast_toolkit() if tool is None else tool
    result = toolkit.trace_lint(present, specs_dir=str(specs_dir))
    grounded = broad_ground_truth(specs_dir)

    violations = [
        finding
        for finding in result["findings"]
        if finding["rule"] == DANGLING_RULE
        and is_test_symbol(finding["name"])
        and finding["detail"] not in grounded
    ]
    return sorted(violations, key=lambda f: (f["file"], f["line"], f["detail"]))


def _display_path(raw: str, repo: Path) -> str:
    try:
        return str(Path(raw).resolve().relative_to(repo))
    except ValueError:
        return raw


def main(argv: Sequence[str] | None = None) -> int:
    ap = argparse.ArgumentParser(
        description=(
            "Spec-authority gate (RUL-013): fail when a @traces decorator on a "
            "test cites a requirement id that no spec defines."
        )
    )
    ap.add_argument(
        "--roots",
        nargs="*",
        default=None,
        help=f"Roots to scan (default: {' '.join(DEFAULT_ROOTS)})",
    )
    ap.add_argument(
        "--specs-dir",
        default=DEFAULT_SPECS_DIR,
        help=f"Spec corpus providing the ground truth (default: {DEFAULT_SPECS_DIR})",
    )
    ap.add_argument(
        "--max",
        type=int,
        default=0,
        help=(
            "Fail if the ungrounded-id count exceeds this (ratchet ceiling; "
            "pin 0 once the known fabrications are resolved)."
        ),
    )
    ap.add_argument("--json", action="store_true", help="JSON output.")
    args = ap.parse_args(argv)

    repo = repo_root()

    def _resolve(rel: str) -> Path:
        path = Path(rel)
        return path if path.is_absolute() else repo / path

    root_names = DEFAULT_ROOTS if args.roots is None else tuple(args.roots)
    roots = [_resolve(r) for r in root_names]
    specs_dir = _resolve(args.specs_dir)

    violations = find_violations(roots, specs_dir)
    rows = [
        {
            "file": _display_path(v["file"], repo),
            "line": v["line"],
            "name": v["name"],
            "req_id": v["detail"],
        }
        for v in violations
    ]

    if args.json:
        print(json.dumps({"violations": rows, "count": len(rows), "max": args.max}))
    else:
        print(
            f"Spec-authority gate (RUL-013): {len(rows)} test-cited requirement "
            f"id(s) defined by no spec; ceiling --max {args.max}."
        )
        for row in rows:
            print(
                f"  {row['file']}:{row['line']}: {row['name']} cites "
                f"{row['req_id']} — defined by no spec under {args.specs_dir}/"
            )

    if len(rows) > args.max:
        print(
            f"FAIL: {len(rows)} ungrounded requirement id(s) cited by tests "
            f"exceeds the allowed {args.max}. Specs are authoritative "
            "(RUL-013): write the requirement into the spec, then cite it — "
            "never invent an id to satisfy a test.",
            file=sys.stderr,
        )
        return 1
    print(
        "PASS: every test-cited requirement id is grounded in a spec.", file=sys.stderr
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())