Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Gates › affected-selection

Gate: affected-selection (RUL-040)
Purpose Run only the tests a change can affect, and say by name why when a change cannot be bounded
Where it runs scripts/run_tests.py --affected <range> step 0.75, called by scripts/hooks/pre-push and by deploy/ship.sh verify
Trigger Every push with committed changes against origin/master; every ship
Exact command .venv/bin/python scripts/run_tests.py --affected "origin/master...<sha>" (hook); the selector alone: python scripts/affected_tests.py --since <rev> [--json] or --files <path>...
Blocks A refused verdict aborts the run ("affected-refusal") naming each missing fact, unless --affected-decision <citation> records a human decision. bounded and named_full run the resolved subset
Ruling / ADR RUL-040 (Enforcement: gate scripts/affected_tests.py RUL-040, gate scripts/run_tests.py RUL-040); RUL-029; RUL-032; ADR-0118

Why it exists

RUL-040 (Mohamed, 2026-10-20): "never escalate=true is meaningless, it must be measured never the whole suite that is a huge bug", and "LLM doesn't determine that, deterministic code through blast radius". The selector's own docstring records why: "2026-08-11: a 2-file deploy/hook change escalated to the full 32k suite because these edges were invisible". Before that, the 2026-09-21 ruling: "No test that passes must run twice ... If the hook escalates to a full run then the logic is wrong." 2669f4882 (2026-10-21) removed the boolean escalate arm: the selector emits a verdict row per changed file.

How a change is resolved

scripts/affected_tests.py (module docstring, AST toolkit) combines three sources:

  1. The import closure from plans/ast-ultimate.json (src) and plans/ast-tests.json (tests): for a changed src/ module, walk importers backwards to depth 3 (MAX_DEPTH) and select every test module that imports any of them.
  2. Reference edges (plans/reference-edges.json, from scripts/extract_reference_edges.py): subprocess argv literals (how hook tests bind to .claude/hooks/*.sh), pytest-bdd scenario bindings, Prolog consult(...) paths, and reads of module-level path constants.
  3. The manifest (plans/affected-tests-manifest.json): only dynamic leftovers, plus the scripts/X.py -> tests/scripts/test_X.py stem rule.

The floor tests/infrastructure always runs (SMOKE_FLOOR).

Per-file verdicts. Each changed file gets a row: resolved (with the edges that selected tests), inert (documentation with zero reference edges: smoke floor only), named_full (a change whose honest blast radius is the whole suite, recorded by name), or refused (no resolution). A changed file outside src/ and tests/ with no edges and no manifest coverage is an untested-file defect to report, not a reason to run the suite (_UNTESTED_IS_NOT_A_FULL_SUITE). The run-level verdict is bounded, named_full or refused.

Plans in run_tests.py (_plan_affected_run, AST toolkit):

Verdict What runs Report
bounded the resolved subset "subset run — non-authoritative, full suite still owed"
named_full still only the resolved subset; "The whole suite is NEVER auto-run — the name is the verdict" named_full lists each path and reason
refused, no decision nothing: abort the refused reasons, named
refused, with --affected-decision the bounded remainder the decision and the refusal rows
every row inert nothing beyond passing (GATE-AFFECTED, 68024751d, 2026-10-21): infra, gates and smoke floor are skipped "all changes inert (non-code)"

Diagram projects-orchestrata-ultimate-gates-affected-selection-1

Real output

The item-3 resolution of 2026-10-22 (affected-item3.json in the session scratchpad): verdict bounded, with rows such as .claude/hooks/inject-codebase-map.sh: resolved via 1 reference edge(s) (tests/scripts/test_session_start_hooks.py) and docs/CODEBASE_MAP.md: documentation with zero reference edges — certified inert, smoke floor only. The run that followed selected 134 tests in 7 modules.

Typical failures and how to fix them

Failure Cause Fix
REFUSED ... outside the serialized import closure plans/ast-ultimate.json lacks a new src/ file, for example after restoring the committed snapshot (improvement log F3, D14) Refresh the snapshot before the run; never restore a generated input between measuring and dispatching
origin/master not found — cannot bound the blast radius no base in this checkout Fetch, or pass --affected-decision with a citation
file or directory not found at collection the selection kept deleted test paths transferred to codetools (375509928, 2026-10-22: 70 modules did not run) Fixed: every absent path now leaves the selection
Refusal on .opencode/*, empty __init__.py, build scripts fixed in 306e9eb5e (2026-10-21): they are inert —
A named_full row a dependency, env or root-conftest change Read the name; run more only if the blast radius requires it

Open against the ruling. RUL-040 asks for method-level blast radius and a function-level reuse ledger. In this tree the reuse ledger is module-level (scripts/test_green_ledger.py docstring); codetools has a select-affected verb (8f82f29f, 2026-10-20) that RUL-040's Stage 0 chose as the seam (e7570a88d). Whether orchestrata's pre-push consumes that verb today is not measured.