Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › p-rul040-fail-closed-blast-radius

P-RUL040 — reuse ledger + blast radius accurate: fail-closed selection, never a silent full run

Source: plans/briefs/p-rul040-fail-closed-blast-radius.md at 8a2cb42ad · last changed 2026-10-20 · first committed 2026-10-20 · 147 lines.

Prose pending.

Plan text

P-RUL040 — reuse ledger + blast radius accurate: fail-closed selection, never a silent full run

Queue position 2 (handoff 2026-10-20 §3). Task ledger: the P-RUL040 entry. Contract: docs/ast-checkable-rules.md §4 (the ruled mechanism: symbol-precise blast radius + fail-closed "cannot bound, named set, explicit decision — never escalate to a full run") + ADR-0118's adjudication that rts.py is a MERGE-ADOPTION into codetools' richer siblings, not a file move.

This is a TWO-STAGE brief. Stage 0 is measurement + proposals + an adoption map — NO code. The coordinator rules the seam and the adoption from your numbers; Stage 1 fences are named in the brief amendment AFTER the ruling. Do not write implementation code in Stage 0.

RULING INJECTED 2026-10-20 (Mohamed, verbatim) — binds the Stage 1 mechanism shape

"never escalate=true is meaningless, it must be measured never the whole suite that is a huge bug"

"the code in history has a shared ledger, it should be at the function level and the blast radius should be at the method level with blast radius"

Two granularity rulings, and they are the acceptance criteria for Stage 1, not preferences:

  1. The reuse ledger keys at FUNCTION level. Measured current state: scripts/test_green_ledger.py keys on MODULE content (ledger.partition(all_modules, repo_root=…, path=…); reports/test-green-ledger.json, 556,595 bytes at 2026-10-20 05:22), so one changed line in a 60-test module invalidates all 60 greens. A function-level key reuses every test function whose own body and whose own measured dependencies are unchanged. This is the "ledger-precision half" §"The measured landscape" below already names at line 43.
  2. The blast radius is computed at METHOD level. Measured current state: scripts/affected_tests.py is module-granularity (its own docstring says so), which is why rung 1 symbol closure exists but is not what the module-level path uses. The radius must resolve a changed def/method to the test FUNCTIONS that reach it — scripts/rts.py already hashes every def/class at any depth plus module residue, and foundation/symbol_snapshot.py already carries body_hash/residue_hash, so this is an adoption of machinery that exists, not new invention.
  3. escalate: true is not a verdict and must not survive as one. It is a boolean that discards the measured reasons beside it and buys a 30k-test run. Rung 4's NAMED full verdicts (dependency-or-environment-change:<path>) and rung 5's SelectionRefused naming the missing fact are the replacements; the consumption arm at elif resolved.get("escalate"): in run_tests.py (cite by symbol — the concurrent T21/D7 packet moved it off line 1446 to ~1513) must go.

Cost of the status quo, measured this session (2026-10-20): a root-tests/conftest.py change produced escalate: true whose own reason text read "for the ROOT conftest that IS the whole suite; this is the honest blast radius". Symbol-level measurement of the same change contradicts it — the provenance block defines NO symbol any test imports (grep for _ou_provenance|_expected_pkg|_actual_pkg|_direct_url_raw| _actual_origin|_expected_origin|_PROVENANCE_DIST across tests/ outside conftest: zero hits), so its true radius is "module-level session-start code, proven executed by any module that collects", i.e. one representative module per lane. The oracle meanwhile DID name the two real edges precisely (.envrc → tests/deploy/test_envrc_toolchain.py, pyproject.toml → tests/core/test_pyproject_contract.py) — so the precision exists at file granularity and is thrown away by the escalate boolean. Stage 1 must make the ladder's rungs 1–5 the ONLY outputs.

The founding incidents

  1. The silent escalation: scripts/run_tests.py's affected-selection escalates to a FULL suite run when it cannot bound a change — consumption point measured at the elif resolved.get("escalate"): arm (~line 1513 post-D7; cite by symbol, lines drift). RUL-040-to-be bans this as a silent path: an unboundable change must produce a REFUSAL naming the unboundable set + requiring an explicit recorded decision — never a quiet 18-minute full run that looks like diligence.
  2. The dead run 906142's 181s tests/bdd spec_012 affected-scope failure (L23 triage): the affected machinery's own cost/accuracy class, assigned to this packet by the triage.

The measured landscape (Stage 0 inputs — verify each, do not trust)

Stage 0 deliverables (report; no code)

M1 — the adoption map. Table: every rts.py behavior (each fact predicate, the byte-hashing contract, the safety escape, the select ladder) → the codetools rung/module that already owns it, or GAP (unique residue). Candidate conclusions to cost, not assume: is rts' unique residue only the standalone swipl CLI + the fact-schema interop surface? Does codetools' rung-2 data_loaders + rung-4 named-full already subsume the safety escape? Cite file:line on both sides.

M2 — the seam proposal. How should orchestrata's run_tests.py --affected consume fail-closed selection? Cost each (lines of change, new dependencies, failure modes, latency): (a) codetools as an orchestrata dev-group path/git dependency, importing deploy.affected; (b) the deployed codetools CLI as a subprocess with a JSON contract; (c) fact-file interop — orchestrata emits the diff/snapshot facts (rts' frozen sym_hash/2 + file_hash/2 schema), codetools' ladder consumes; (d) port the REFUSAL SHAPE into affected_tests.py natively (the ladder stays codetools'; orchestrata grows the named-set refusal

M3 — the escalation anatomy. Trace resolved.get("escalate"): who computes resolved (affected_tests.py? which function), every input that sets escalate, every consumer of the full-run decision, and the exact points where a refusal-with-named-set could replace it. Include the ledger interaction: what does a refused selection record (nothing? a named-refusal row?).

M4 — the ledger-precision half. What would symbol-granularity reuse keys require: measure the current ledger key shape (run_tests.py's reuse mechanism — module + content hash?), and what §4's "module + import closure at symbol granularity" adds; cost it against the body_hash/residue_ hash data ALREADY in plans/ast-ultimate.json and codetools' symbol_snapshot. Recommend a phase split if the full precision is a bigger packet than the escalation fix.

M5 — the 181s spec_012 case. Reproduce the shape from the dead-run evidence (reports/ + L23): what made the bdd affected-scope selection cost 181s; does codetools' rung-3 dynamic-module handling (bdd scenarios()) already answer it; what orchestrata's selector lacks.

Stage 1 — RULED 2026-10-20 (coordinator adjudication appended to reports/p-rul040-stage0.md)

Stage 0 delivered and adjudicated: M1 adoption map accepted (citations spot-verified); M2's option-(c) recommendation INVALIDATED by measurement (its "no new dependencies / codetools already on sys.path" premise is false — zero codetools imports in the tree, no installed package, bin dirs are not import paths). Seam ruling: option (b) — a codetools select-affected verb consumed as a subprocess JSON contract from the DEPLOYED CLI (ADR-0117's consumption doctrine; a concrete consumer now exists), per the adjudication's 5-point shape: verb contract (exit 0 bounded / exit 2 refusal-with-named-set / other non-zero tool error), orchestrata consumption in affected_tests.py + the run_tests.py escalate-arm replacement (refusal → abort with named set + explicit-decision citation mechanism; tool-missing → fail-closed naming the tooling state — NEVER a silent full run), rts.py extract's fact schema frozen as the interop surface (<100 LOC JSON transform), rts select RETIRED at verb landing, option (a) rejected (cross-repo import dependency = a boundary ruling not made), option (d) rejected (ladder drift). Phase split adopted: Phase 1 = the refusal seam (the 2026-10-20 push-gate incident — 39,102-test named-full escalation collapsing into 1,957 ambiguous "batch error" lines — joins the founding regressions: gate output must NAME the escalation cause and budget state); Phase 2 (deferred, separately measured) = symbol-granularity ledger keys.

Stage-1 fences (two repos, sequenced — codetools verb FIRST, deploy, then orchestrata consumption): codetools: new files in the deploy lane + the verb wiring (coordinator serial-integration step for pyproject/console/deploy per ADR-0118 §4); orchestrata: scripts/affected_tests.py, scripts/run_tests.py (the escalate arm only), scripts/rts.py (select retirement + the JSON transform), new tests both sides. A Stage-1 worker brief is minted from this section + the adjudication when dispatch slots open (post-push, post-P-RUL039 — run_tests.py fence collision with P-RUL039 must serialize).

Report

M1-M5 with every claim carrying file:line on both repos, the two recommendations (seam, phase split) with arguments, anything you could not measure named. Read-only Stage 0: the ONLY file you write is your report (reports/p-rul040-stage0.md). RUL-037 applies to measurement too — no python -c/heredocs; use the AST toolkit, jq, grep.