Orchestrata Ultimate › Gates › test-phase-cap
Gate: test-phase-cap (RUL-038 amendment)
| Purpose | Bound every test phase in time, count what it will run before it starts, and refuse unsanctioned multi-file pytest sessions |
| Where it runs | scripts/run_tests.py step 3 (every run, including --affected from the pre-push hook and ship); the observer plugin pytest_live_observer in every pytest a Claude session starts |
| Trigger | The test phase, after preflight and gates; any pytest collection |
| Exact command | none of its own; the cap is ORCH_TEST_CAP_SECONDS (default 300); --solo-retry-cap N (default 25) |
| Blocks | A capped run exits non-zero with CAPPED "regardless of failure count"; a multi-file pytest session without the sanctioned-runner stamp exits at collection with rc 4 and 0 tests |
| Ruling / ADR | RUL-038, amendment of 2026-10-19/20 ("SELECTED, COUNTED and CAPPED"); RUL-009 (observed runs) |
The ruling
Recovered from the Claude session transcript 3a241b76 (lines 2659 and 2678) and recorded
on the RUL-038 page: "a HARD 300 s cap on the test phase (preflight and gates excluded).
At the cap, stop all workers, print run/left/failed, record the modules that passed into
the green ledger, and exit non-zero with CAPPED." And on 2026-10-19: "any more than 1
test file, detect it and stop immediately".
What it does
- Selected and counted. Before tests start,
run_tests.pyprintsSelected: N tests in M modules (K modules reused from ledger) · hard cap 300s, then a liverun X / selected Y · left Z · failed Fline (ProgressTicker). - The clock starts after preflight and gates (
_main, step 3 comment)._test_cap_secondsreadsORCH_TEST_CAP_SECONDS, default"300";_run_deadlineturns it into a deadline. - Every child carries the stamp.
_child_run_env(deadline)setsORCHESTRATA_TEST_DEADLINE,ORCHESTRATA_SANCTIONED_RUNNERand the run id; at the deadline each child pytest kills itself (arm_deadline), so the cap needs no cooperation from the executor's dispatch loop. - At the cap every module that passed goes to the green ledger (the resume point), and
the run prints
⛔ CAPPED at the 300s test-phase cap (RUL-038)and exits non-zero. - Multi-file refusal.
pytest_live_observer.pytest_collection_finish(AST toolkit) counts the files behind the session's selected items; with more than one file and noORCHESTRATA_SANCTIONED_RUNNER, it callspytest.exit(..., returncode=USAGE_ERROR). The observer loads in every pytest a Claude session starts, because~/.claude/settings.jsonexportsPYTEST_ADDOPTS=-p orchestrata_pytest_observer. - Solo retry. Modules that failed in a batch re-run, each alone, up to
--solo-retry-cap(default 25, raised from 5 on 2026-08-07 because "the cap ran backwards: the more modules failed, the less the run checked"). A module that passes alone is a load or order flake, logged to.flake_recoveries.ndjson; over the cap modules are left UNVERIFIED, not judged (argparse help, AST toolkit).
Real output
Attempt 3 of the 2026-10-22 item-3 push (session scratchpad push-item3-retry2.log):
[run_tests] Selected: 134 tests in 7 modules (0 modules reused from ledger) · hard cap 300s
[run_tests] run 48 / selected 134 · left 86 · failed 0
...
[run_tests] 🔁 1 module(s) failed in batch — re-running each in an isolated process (solo-retry cap 25)
[run_tests] ❌ 133 tests passed, 1 failed, 0 errors, 0 skipped · 6/7 modules in 121s (2.0m)
[run_tests] 🔁 solo retry: 1 batch-failed module(s) re-run in isolation — 0 recovered (load/order flakes), 1 still failed · COST 12 test(s) executed twice, 31.7s serial tail
Typical failures and how to fix them
| Symptom | Cause | Fix |
|---|---|---|
CAPPED |
the selection does not fit in 300 s | Re-run: the ledger resumes from what passed. Narrow the selection; never raise the cap to pass |
| Every batch child fails at collection with a RUL-038 refusal | children launched without the stamp. On 2026-10-20 the batch execution path built its own env with only the deadline: 1,946 guard kills on one push, 100% of the FAILED lines (167d6b888) |
Fixed: both call sites compose {**os.environ, **_child_run_env(deadline)} |
A plain pytest tests/ gate fails from a Claude session and passes from a terminal |
the observer refuses the multi-file session (ship's wiki gate, 13 files) | Fixed in 7b859bb4a (2026-10-22): ship's gate_pytest stamps ORCHESTRATA_SANCTIONED_RUNNER=ship.sh |
| Multi-file pytest refused in another repo | PYTEST_ADDOPTS loads the observer everywhere (session notes, 2026-10-22, hit in the hermes repo) |
Run through the repo's sanctioned runner, or one file per pytest |
| A test module outruns the cap by itself | for example tests/deploy/test_install_tools.py, where every test ran the full installer |
7309a3488 (2026-10-22) shares one read-only install among the tests that only read it |
History. The cap work was adopted on 2026-10-20 (4a470482c) with three fixes beyond
the prior session's diff: the default was an invented 120 s and became the ruled 300 s; a
capped run with zero failures had exited 0 (a false green), now decoupled; and
_child_run_env had fabricated its env keys (ORCH_TEST_DEADLINE against the declared
ORCHESTRATA_TEST_DEADLINE, and no sanctioned-runner stamp), so "the deadline could never
reach" a child. The fabricated-key class became test-review rule T21 (25f06c3f6).