Knowledge Wiki Orchestrata Ultimate

RUL-040 — verification scope is measured at symbol granularity; a full-suite escalation is a bug

Ruling

Mohamed, 2026-10-20, verbatim:

"never escalate=true is meaningless, it must be measured never the whole suite that is a huge bug"

"the code in history has a shared ledger, it should be at the function level and the blast radius should be at the method level with blast radius"

And, on the behaviour that prompted it — the coordinator had run a 401-module full tree in codetools and then a full orchestrata suite because an oracle returned escalate: true:

"no, if the code didn't change why run the same test again?"

"LLM doesn't determine that, deterministic code through blast radius"

Three obligations follow:

  1. escalate: true is not a verdict. A boolean that discards the measured reasons computed beside it, and buys a ~30k-test run, is a defect. An unboundable change produces a NAMED verdict (dependency-or-environment-change:<path>) or a REFUSAL naming the missing fact and the set it could not bound — never a quiet full run that looks like diligence.
  2. The blast radius is computed at METHOD level. A changed def/method resolves to the test FUNCTIONS that reach it. Module-granularity import closure is the coarse fallback, not the answer.
  3. The reuse ledger keys at FUNCTION level. One changed line in a 60-test module must not invalidate 60 recorded greens; a test function is reused when its own body and its own measured dependencies are unchanged.
  4. Scope is decided by deterministic code, never by the model's judgment. The coordinator queries the oracle and obeys it. "This might touch everything" is a heuristic where a derived answer was available, and is not a justification.

Evidence

Measured 2026-10-20 in orchestrata-ultimate (branch components) and orchestrata.codetools:

Enforcement

Judgment-only at recording time (2026-10-20). This is a recorded gap, not the intended end state — RUL-040 is explicitly NOT a judgment ruling, it demands deterministic selection. It is declared none only because scripts/rulings_lint.py verifies a declared gate actually carries the rule-id, and no gate in the tree references RUL-040 yet. Each mechanism below replaces this line with its gate entry when P-RUL040 Stage 1 lands, and the ruling is complete only when all four are in force.

Mechanism (owed, RUL-014)

Compliance of each candidate gate, measured 2026-10-20, stated honestly rather than implied:

  1. scripts/rts.py — COMPLIES in substance, not yet wired. The method-level blast radius the ruling requires already exists: it hashes every def/class at any depth plus module residue (<mod>.<module>), drives tabled selection under swipl, and forces FULL-RUN on escapes. It is not the default path behind run_tests.py --affected; ADR-0118 adjudicates it a MERGE-ADOPTION into codetools' richer siblings, not a file move.
  2. scripts/affected_tests.py — PARTIALLY COMPLIES, and is the open violation. It computes bounded sets with per-file named reasons and already certifies inertness mechanically ("documentation with zero reference edges — certified inert, smoke floor only"), but it still emits escalate: bool, and its own docstring records the pathology ("a 16-test change escalated to 33,104 tests").
  3. scripts/test_green_ledger.py — PARTIALLY COMPLIES. The RUL-032 reuse ledger exists and works ("♻ RUL-032: 43 module(s) unchanged since their last green run — reused, not re-run") but keys on MODULE content (ledger.partition(all_modules, …); reports/test-green-ledger.json, 556,595 bytes), which this ruling requires at FUNCTION granularity.
  4. scripts/run_tests.py — VIOLATES. The consumption arm elif resolved.get("escalate"): (cite by symbol; the concurrent T21/D7 packet moved it off line 1446 to ~1513) prints "escalated to the FULL suite" then asserts "this IS the authoritative run". Stage 1 removes it.

Status: OPEN VIOLATION until P-RUL040 Stage 1 lands. The packet (plans/briefs/p-rul040-fail-closed-blast-radius.md) carries this ruling at its head, dated and quoted verbatim, as the Stage 1 acceptance criteria: rungs 1–5 of docs/ast-checkable-rules.md §4 become the selector's only outputs, the escalate arm goes, and the ledger re-keys to test-function granularity.

Companion rulings this one sharpens: RUL-029 (scope verification by query — this ruling specifies the granularity the query must reach, and bans the whole-suite answer), RUL-032 (tests run once per code state — this ruling moves "code state" from module content to function content plus measured method-level dependencies).

Verification