Orchestrata Ultimate › Gates › rehearsal-exclusion

Gate: rehearsal-exclusion

At a glance


Purpose	Keep tests that exercise real deploy surfaces out of the promotion and push gates. Mohamed, 2026-10-19: "promotion gate = functionality tests only; rehearsal-class (real deploy/ship/incus/terraform/tofu/load) tests exist but NEVER in the gate."
Where it runs	`scripts/run_tests.py::_main`, after the green-ledger path is computed and before `ledger.partition` (the RUL-039 block). Producer: `scripts/derive_test_categories.py`.
Trigger	Gate contexts only: `--affected` is set, or the requested paths are exactly `["tests/"]`; and `--include-rehearsal` is not set (fix `5c2de2bbe`, 2026-10-21).
Exact command	Consumer: any `python scripts/run_tests.py --affected <range>` or `python scripts/run_tests.py`. Producer: `python scripts/derive_test_categories.py` (write), `--check` (recompute and exit non-zero on drift), `--stdout` (table).
Blocks	A missing, unparsable or `derivation_version != 1` artifact aborts the run at stage `rehearsal-artifact`, printing `✗ <reason> — regenerate with python scripts/derive_test_categories.py`. Otherwise it removes modules and prints `🎭 rehearsal-class excluded: N module(s)`.
Ruling / ADR	RUL-039; siblings RUL-038 / ADR-0111 and ADR-0110; category derivation per RUL-036.

Why it exists

RUL-039's founding incident: dead run 906142 (2026-10-19) had 20 reds, 14 of them rehearsal-class deploy tests pulled into a gate run by an over-wide blast radius — 6 × tests/cli/test_cli*.py deploy install/verify assertions, 5 × codetools verify_venv_sync DeployError, 3 × codetools PrologGatesError (docs/lessons/L23-exoneration.md, cited on the ruling page).

What it checks

The producer — `scripts/derive_test_categories.py`

Module docstring (AST toolkit): the category "is COMPUTED from what a test actually does — never hand-set … a label always carries its file:line evidence". A test module is rehearsal when it has one of these markers:

argv-literal: a subprocess or CLI argv token incus, lxc, docker, terraform or tofu in execution position (_ARGV_TOKENS) — not prose, docstrings or asserted-against data;
env-gated / pytest-marker: a guard on INCUS_HARNESS_IT (_LIVE_ENV_KEYS) or a @pytest.mark.<…>_live / @pytest.mark.live marker;
deploy-surface: a subprocess call into one of the repo's deploy scripts (script names with .sh, never a bare "deploy" word).

Deliberately not markers: importing a deploy module to unit-test pure functions, tmp_path deploy configs, "deploy" in a test name.

The artifact reports/rehearsal-class-modules.json (read 2026-10-23): derivation_version: 1, generated_at: 2026-10-21T06:31:49Z, a tests_tree_fingerprint, and 16 rows:

Module	Marker
`tests/claude/test_unshipped_guard.py`, `tests/claude/test_unshipped_guard_coverage.py`	deploy-surface `ship.sh`
`tests/deploy/test_deploy_assets.py`, `tests/deploy/test_install_gitlab_runner_executor.py`	deploy-surface `install-gitlab-runner-executor.sh`
`tests/deploy/test_deploy_script.py`	deploy-surface `provision_environments.sh`
`tests/deploy/test_install_orchestrata_{cc,secrets,toolchain}.py`	deploy-surface `install-orchestrata.sh`
`tests/deploy/test_restart_tool_instances.py`	`restart-tool-instances.sh`
`tests/deploy/test_retire_plain_units.py`	`retire-plain-units.sh`
`tests/deploy/test_ship_merge_master.py`	`ship-merge-master.sh`
`tests/gateway/test_mcp_registration_installer.py`	`install-claude-config.sh`
`tests/harness/test_incus_isolation.py`, `tests/rehearsal/test_enroll_orchgraph_show.py`	env-gated `INCUS_HARNESS_IT` + marker `incus_live`
`tests/harness/test_isolation.py`	argv-literal `incus`
`tests/scripts/test_check_infra_drift.py`	marker `incus_live`

The rows' at evidence paths point into /home/mohamed/workspace/wt-COMPONENTS, so this copy of the artifact was generated in that worktree and carried over.

The consumer — the RUL-039 block in `run_tests.py`

Read with the AST toolkit:

An explicitly named rehearsal .py file still runs ("a deliberate rehearsal").

Gap between ruling and code (measured 2026-10-23). RUL-039's Enforcement text says a "fingerprint-mismatched artifact fails closed". The runner's _rehearsal_modules() checks existence, JSON parse and derivation_version only; it does not compare tests_tree_fingerprint with the current tree. Drift is caught only by derive_test_categories.py --check, which "recomputes and compares against the committed artifact, exiting non-zero on drift" (its docstring). A search of the tree's shell, YAML, TOML and JSON files found no hook, CI job or ship stage that runs --check, so a new rehearsal-class test is excluded only after someone regenerates the artifact.

History

35d039707 (2026-10-21) — P-RUL039 W2+W3: the exclusion, the 🎭 line and --json rehearsal_excluded, the --include-rehearsal override, fail-closed on a missing or stale artifact; 14 derivation tests plus test_run_tests_rehearsal_exclusion.py.
5c2de2bbe (2026-10-21) — regression fix: the exclusion and its fail-closed check applied to every run, so a deliberate subset run with no artifact (test_main_mixed_reuse with REPO_ROOT=tmp_path) aborted. Now gated on args.affected is not None or requested_paths == ['tests/'].

Typical failures and how to fix them

Symptom	Cause	Fix
`✗ rehearsal-class artifact missing — regenerate with python scripts/derive_test_categories.py` and the run aborts	No artifact in this tree (a fresh tree, or a sandboxed `REPO_ROOT`)	Run the producer and commit the artifact
`… derivation_version mismatch`	The artifact was written by a different producer version	Regenerate
A deploy test still runs in a push gate	It is not in the artifact (new test, stale artifact), or the push named it explicitly	`python scripts/derive_test_categories.py --check`; regenerate when it reports drift
A rehearsal test is needed on purpose	—	Name the file explicitly, or pass `--include-rehearsal`; never in a gate run