Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › p-rul039-rehearsal-exclusion

P-RUL039 — the promotion gate never runs rehearsal-class tests (derived labels, fail-closed consumption)

Source: plans/briefs/p-rul039-rehearsal-exclusion.md at 8a2cb42ad · last changed 2026-10-20 · first committed 2026-10-20 · 126 lines.

Prose pending.

Plan text

P-RUL039 — the promotion gate never runs rehearsal-class tests (derived labels, fail-closed consumption)

Queue position 1 (handoff 2026-10-20 §3). Task ledger: the P-RUL039 entry (registered this session). Mechanism design surface: docs/ast-checkable-rules.md §4 + §3#13 — category DERIVATION feeds the gate exclusion; the category is COMPUTED from what a test depends on, never hand-set (RUL-036's own words). RUL-039 wiki registration rides the landing (rulings_lint fails a declared-but-missing mechanism) — the coordinator writes the page; step W0 below recovers the ruling's verbatim text first.

The founding incident (what this packet makes impossible)

Dead run 906142 (2026-10-19, triaged in docs/lessons/L23-exoneration.md): 20 reds, of which 14 were REHEARSAL-CLASS tests — real deploy surfaces dragged into a gate run by an exaggerated blast radius: 6× tests/cli/test_cli*.py deploy install/verify assertions (real-machine surfaces, 1-4s assertion failures), 5× codetools verify_venv_sync DeployError, 3× codetools PrologGatesError. The class Mohamed ruled (same day) must never run in the promotion gate. ADR-0111/RUL-038 established the sibling doctrine for container-CREATING tests (they leave the test tree for rehearsals/); RUL-039 covers the gate-exclusion of the rehearsal CLASS (real deploy/ship/incus/terraform/tofu/ load surfaces) that still lives in tests/ — the RUL-038 wiki page names this repo's violators: the incus_live lane and the INCUS_HARNESS_IT=1 round-trip in tests/harness/test_incus_isolation.py.

Fence

scripts/derive_test_categories.py                  (NEW — the derivation pass)
reports/rehearsal-class-modules.json               (NEW — committed artifact, RUL-030)
scripts/run_tests.py                               (selection exclusion + report + fail-closed)
tests/scripts/test_derive_test_categories.py       (NEW)
tests/scripts/test_run_tests_rehearsal_exclusion.py (NEW)

READ-ONLY for measurement: src/orchestrata_ultimate/promotion/promoter.py (the test verb at :214 → run_tests.py --json), scripts/affected_tests.py, tests/cli/ + tests/harness/ (the founding-class exemplars), docs/ast-checkable-rules.md §3#13 + §4, the RUL-038 wiki page. scripts/run_tests.py note: it just landed T17's D7 edits (constants _RUN_ID_ENV/ _HEAD_COMMIT_ENV imported) — do not regress them; your selection edits are a different region.

W0 — verbatim ruling recovery (report, do not publish)

Find Mohamed's 2026-10-19 ruling text that founded this packet (search context.json session records around the dead-run triage and reports/errors-and-lessons-2026-10-19.md M8/L23 citations). Report what you find verbatim + cite; if no verbatim exists beyond L23's paraphrase ("the class Mohamed ruled (same day) must never run in the promotion gate"), say exactly that — the coordinator registers the wiki page with what exists, never with an invented quote.

W1 — the derivation pass (scripts/derive_test_categories.py)

Computes, per test module under tests/, a category row. This packet derives ONE class: rehearsal. Schema (designed to EXTEND, not be replaced, when RUL036-DERIVE (#571) adds the full functional/integration split):

{"derivation_version": 1,
 "generated_at": "...",
 "tests_tree_fingerprint": "<sha of the sorted (module_path, module_body_hash) list>",
 "rows": [{"module": "tests/harness/test_incus_isolation.py",
           "category": "rehearsal",
           "markers": [{"kind": "argv-literal", "value": "incus", "at": "file:line"}, ...]}]}

Derivation signals (AST-based — reuse the repo's ast machinery patterns; every row carries its file:line evidence, a label without evidence is invalid):

  1. subprocess/CLI argv string literals whose tokens include incus, lxc, docker, terraform, tofu (the RUL-038/ADR-0111 marker set) in EXECUTION position (Popen/run/check_call/os.system argv, or shell strings passed to them) — not prose, not docstrings, not asserted-against data.
  2. env-gated live lanes: module-level or fixture-level guards on INCUS_HARNESS_IT-style live-run env keys (measure the actual key set from tests/harness/).
  3. real deploy-surface invocations: subprocess calls into deploy/, ship, promote, installer scripts against real targets (measure how tests/cli/test_cli*.py actually invoke — the marker must be the mechanism, not the directory name; a directory-name rule is hand-setting in disguise and is FORBIDDEN). Deliberately NOT markers: importing deploy modules to unit-test their pure functions; tmp_path fixture deploy configs; the word "deploy" in test names/docstrings. Output: the committed artifact + a --check mode (recompute, compare against the committed artifact, exit non-zero on drift — the staleness gate consumes this) and --stdout human table.

W2 — gate consumption (scripts/run_tests.py)

  1. Selection (ALL batched/gate contexts — the promotion verb --json, --affected, plain suite runs) EXCLUDES modules whose committed artifact row says rehearsal.
  2. NEVER silent: the run report gains an exclusion line — rehearsal-class excluded: N modules (named list at reports/rehearsal-class-modules.json) — and --json output carries the named set. An exclusion nobody can see is the failure mode this packet exists to kill.
  3. Deliberate runs stay possible: naming a rehearsal module EXPLICITLY on the command line (or --include-rehearsal) runs it — rehearsal semantics per ADR-0111 ("run deliberately when that code changes"). Explicit naming overrides the exclusion and says so in the report.
  4. FAIL-CLOSED staleness: artifact missing, unparsable, derivation_version mismatch, or tests_tree_fingerprint not matching the current tree → gate-mode runs REFUSE with the named state and the regeneration command (python scripts/derive_test_categories.py), exit non-zero. Never silently run everything, never silently run nothing (RUL-040's philosophy, applied here ahead of its own packet).

W3 — tests (real, no mocks/skips; single-module sanctioned shapes)

VERIFY (report verbatim)

.venv/bin/pytest tests/scripts/test_derive_test_categories.py -o addopts="" -q        # pytest-live
.venv/bin/pytest tests/scripts/test_run_tests_rehearsal_exclusion.py -o addopts="" -q # pytest-live
.venv/bin/python scripts/derive_test_categories.py --stdout                            # the real derived set
.venv/bin/python scripts/derive_test_categories.py --check                             # artifact == tree
.venv/bin/ruff check <fence .py> && .venv/bin/black --check <fence .py>
.venv/bin/python scripts/test-review.py tests/scripts/test_derive_test_categories.py --stdout
.venv/bin/python scripts/test-review.py tests/scripts/test_run_tests_rehearsal_exclusion.py --stdout

Plus: the derived set REPORT — every row with its evidence, checked against the known founding class (tests/harness/test_incus_isolation.py live lanes; the tests/cli deploy modules). A derived set that misses a known member or includes a pure unit-test module is a defect — reconcile and report the reconciliation. Coverage on both new scripts (--cov raw lines, honest gaps).

Done

Reply with: W0 verbatim finding, the derivation rule set as implemented (per marker kind, with the anti-hand-setting argument for each), the FULL derived table with evidence, all VERIFY outputs verbatim, coverage raw lines, the run_tests.py exclusion-report sample output (gate-mode and explicit-override), anything unfaithful named. Workers do not commit. RUL-037. Fence files only. Wiki registration (RUL-039 page + Enforcement line) is the coordinator's step after verification.