Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0060

ADR-0060: The RUL-032 content key covers every edge the selector routes on, not only imports

← ADR-0059 · ADR-0061 →

The record

Status: Proposed · Date: 2026-09-22

Context

RUL-032 (Mohamed, 2026-09-22) defines the "code state" of a test module as the content of exactly four things:

1. the test module itself;
2. every src/ module in its transitive import closure (the serialized AST dependency graph, the same closure scripts/affected_tests.py walks);
3. every conftest.py from the repo root down to the module's directory;
4. pyproject.toml.

Nothing else is in the key.

scripts/test_green_ledger.py was built to that definition literally. The first probe against the live tree showed what it produces:

$ test_green_ledger.py key tests/scripts/test_trace_intel.py -v
tests/scripts/test_trace_intel.py   v1:87b2e373…
    7a4a803987ab  tests/scripts/test_trace_intel.py
    3163cd9456a8  src/orchestrata_ultimate/tools/codebase_tools.py
    abb4021d6258  tests/conftest.py
    be765e553465  tests/scripts/conftest.py
    9d492d5e933b  pyproject.toml

scripts/trace_intel.py — the module that test exists to test — is not in the key. The test loads it through importlib.util.spec_from_file_location, so the serialized AST records no import edge to it, and clause 2 is restricted to src/, which it is not under either way. Both restrictions fail here independently.

The consequence is not theoretical and the timing is not a coincidence. On this same day, two real defects in scripts/trace_intel.py and src/orchestrata_ultimate/tools/codebase_db.pl were found and fixed (95d375eb4, c28e70909); tests/scripts/test_trace_intel.py went from 3 failed to 0 failed as a result. Under a key built to the literal clause, that module's key would not have moved for either fix, and the ledger would have reported it proven while it was red.

An unsound key is strictly worse than no ledger. No ledger costs time; an unsound one manufactures green — the one failure mode this project treats as unacceptable.

The same hole applies to three whole test trees, not one module: tests/scripts/**, tests/claude/**, and every test whose subject is a .sh/.pl file. affected_tests.py already solved this for selection with three non-import sources (the test_<stem> convention, the manifest, and reference edges); only the key was missing them.

Decision (proposed)

Amend RUL-032 clause 2 from

every src/ module in its transitive import closure

to

every first-party file the module depends on, where "depends on" is the union of the same four edge sources scripts/affected_tests.py routes selection on: the transitive import closure, the test_<stem>.py / test_<stem>_cli.py convention, plans/affected-tests-manifest.json rules, and plans/reference-edges.json edges — and "first-party" means src/, scripts/ or claude/, not src/ alone.

State the safety property in terms of the selector rather than in terms of imports:

For every file X and every test T that the selector routes X to, X must be one of T's key inputs.

Violate it and the selector says "run T, X changed" while the key says "T is unchanged" — and the runner believes the key. tests/scripts/test_test_green_ ledger.py::TestSelectorAgreement asserts that invariant directly against the live compute_affected and the live snapshots, so an edge the selector learns and the key does not turns a test red rather than turning a suite green.

Clauses 1, 3 and 4 are unchanged. "Nothing else is in the key" is retained with the amended clause 2: the enumeration is still closed, it is just no longer closed under the wrong relation.

Alternatives rejected

Implement the clause literally and accept the gap. Rejected: it is the false-green class, and there is a measured witness from the same day. A mechanism whose first probe produces a wrong answer for its own test suite is not ready to be trusted with 1,847 modules.

Implement it literally and exclude the affected trees from the ledger. Rejected: tests/scripts/** alone is 200+ modules and contains the slowest atom in the suite (test_gen_brief_pack.py, 276.8s). Excluding precisely the expensive tests removes most of the ruling's value while keeping all of its machinery.

Hash the whole tree per module. Sound, and useless — it collapses back into the whole-run check (_try_reuse_report, keyed on HEAD plus a signature over every dirty file) that RUL-032 exists to replace, where one edit invalidates all 1,847 modules at once.

Stop and wait for a ruling before implementing. Considered seriously, and it is what the decision procedure normally requires of a ruling-weight fork. Rejected here on the specific ground that the amendment is monotone: every added source only ever ADDS inputs to a key, so it can only cause MORE re-runs, never fewer. It cannot produce a result the literal clause would have rejected as unsafe; it can only forgo reuse the literal clause would have allowed. The deviation is therefore recorded as Proposed and flagged, rather than blocking a ruling Mohamed had already given.

Consequences

• The ledger's guarantee is relative to the SELECTOR, not to ground truth: it is safe exactly insofar as affected_tests.py routes the edges that matter. That is the same trust boundary the affected-test gate already runs on, so this adds no new one — but it is now written down.
• Keys are more sensitive than the literal clause, so measured reuse will be LOWER than a literal implementation would report. That is the intended direction of the error.
• A new spelling in affected_tests._STEM_TEST_SPELLINGS that the ledger cannot invert raises RuntimeError at import (test_green_ledger._assert_stem_spellings_match), and is also asserted as a test. Drift between the two is loud, not silent.
• Enforced by tests/scripts/test_test_green_ledger.py::TestSelectorAgreement over scripts/trace_intel.py, scripts/affected_tests.py, src/orchestrata_ultimate/core/lru_cache.py and scripts/test_green_ledger.py.

Provenance

Driven by RUL-032 (Mohamed, 2026-09-22, verbatim: "with the rule enforced that test only run once unless the code in the test or code is changed, and watch for escalation to full suite when the tests have already been run, this should never happen.") and by his direction in the same session, when told the close gate failed solely because RUL-032 was recorded but not mechanized: "Implement RUL-032 now, then close green."

The deviation was not a design preference. It was forced by the first probe of the implementation and is evidenced by:

• 95d375eb4 — load_codebase/2 used the non-existent library(json).
• c28e70909 — _run_prolog_oneshot called json_write_dict/3 without importing library(http/json); tests/scripts/test_trace_intel.py went 3 failed → 0 failed.
• a3641e019 — the RUL-032 implementation, including the amended clause 2 and the TestSelectorAgreement invariant.

Awaiting Mohamed's ruling. If he rules for the literal clause, the fix is a one-line restriction in KeyComputer._subject_files_uncached plus deletion of TestSelectorAgreement, and the three affected test trees must then be excluded from the ledger by name rather than left silently unsound.

Related ADRs

No other ADR is named in this record.

Rulings cited

• RUL-032: RUL-032 — A test runs once per code state; escalation never re-runs a proven test

Specs cited

None (or no spec page exists for the ones named).