RUL-038 — Test runs never create containers; container creation is exercised only by explicit rehearsals

Ruling

Mohamed, 2026-10-19, after asking "why are you creating containers during test run?", verbatim: "do not test container creation in test runds".

• No test run creates, copies, launches or inits a container or VM, through the incus CLI, the Incus REST API, a client wrapper or terraform.
  • A test run is any test-runner invocation over a project's test tree: a developer run, pytest-live, scripts/run_tests.py, a ship gate, pre-commit, or CI.
• Tests may read and exec into the standing fleet containers that already exist.
• Code whose job is creating containers is exercised by explicit rehearsals under rehearsals/, outside the test tree.
  • Examples: codetools' Incus client and gate provisioner, and orchestrata's provisioners.
  • A rehearsal is run deliberately when that code changes, and is never collected by a test run.

Decision record: ADR-0111 (docs/adr/0111-test-runs-never-create-containers-container-creation-is-exercised-only-by-explicit-rehearsals.md).

Evidence

Measured 2026-10-19:

• Six leaked test instances in Incus project orchestrata, the largest 4.1 GiB and six days old.
• With btrfs quotas on t9-btrfs (shared with the prod, test and dev fleets), test-instance deletes blocked test-Redis writes for 99,256 ms and 16,922 ms. A host fsync on the same T9 took at most 24 ms in the same window.
• The bd7aa0dfe ship gate failed on two modules because of those stalls.
• The codetools scratch-provisioner test failed four runs on time alone.

Enforcement

• none (judgment)

Mechanism (RUL-014)

Judgment-only at recording time. Owed (ADR-0111): a static policy gate in the codetools and orchestrata test runs that fails any module under the test tree that calls an instance-creating API. When the gates land, this section names their commits and the Enforcement line moves to gate.

Verification

• Open violations at recording:
  • codetools: tests/gate/test_provisioner_scratch_container.py, tests/incus/live/test_idempotent_ensure.py, the creating scenarios of tests/bdd/test_spec_013.py, and the live case of tests/test_incus_isolation.py.
  • orchestrata: the incus_live lane and the INCUS_HARNESS_IT=1 round-trip in tests/harness/test_incus_isolation.py, plus whatever the census adds.
• Done when:
  • each repo's policy gate runs in its test run and reports 0 instance-creating test modules;
  • a test run in each repo leaves incus list --all-projects unchanged;
  • this page's Enforcement line names the gate commits.

Amendment (2026-10-19/20) — the test phase is SELECTED, COUNTED and CAPPED

Same-day extension of this ruling by Mohamed, recorded post-hoc from the transcript (Claude session 3a241b76, wt-COMPONENTS, lines 2659/2678 — recovered 2026-10-20 by the coordinator session; the adopted code cited "RUL-038 (Mohamed, 2026-10-19): the test phase is SELECTED, COUNTED and CAPPED" before this page carried the text):

• "a HARD 300 s cap on the test phase (preflight and gates excluded). At the cap, stop all workers, print run/left/failed, record the modules that passed into the green ledger, and exit non-zero with CAPPED." (L2659)
• "It prints the selected count before tests start, then a live run X / selected Y · left Z · failed F line. At 5 minutes the run stops with CAPPED, and passing modules are saved to the ledger." (L2678)
• "any more than 1 test file, detect it and stop immediately" (2026-10-19, carried in pytest_live_observer.pytest_collection_finish's docstring): an unsanctioned pytest session selecting tests from more than one file exits at collection (rc=4, 0 tests run); sanctioned runners stamp ORCHESTRATA_SANCTIONED_RUNNER on every child.

Mechanism (landed — repo commit 4a470482c, wt-COMPONENTS branch components)

• scripts/run_tests.py: _test_cap_seconds (default 300, ORCH_TEST_CAP_SECONDS override), _run_deadline, _child_run_env (stamps ORCHESTRATA_TEST_DEADLINE + ORCHESTRATA_SANCTIONED_RUNNER + ORCH_RUN_ID — keys imported from pytest_live_reader's constants, pinned by contract test after a fabricated-key defect was caught at verification), the SELECTED line + ProgressTicker, capped ⇒ exit non-zero regardless of failure count, green-ledger resume point.
• pytest_live_observer.py / pytest_live_reader.py: collection_finished + capped events, multi-file refusal, deadline self-kill (arm_deadline/_hard_stop, CAPPED_EXIT_CODE = 124), count_selected.
• scripts/observe_pytest.py: progress=done/selected (left N) against the SELECTION, never the suite total.
• deploy/ship.sh (7b859bb4a, 2026-10-22): the knowledge-wiki and dr-hannah gate arms run their plain pytest tests/ through gate_pytest, which stamps ORCHESTRATA_SANCTIONED_RUNNER=ship.sh. A Claude session's PYTEST_ADDOPTS loads the observer into ship's pytest too, so before this the wiki's 13-file gate stopped at collection from any Claude session and passed from a terminal. Each arm keeps its own timeout (300 s wiki, 900 s dr-hannah). Pinned by tests/deploy/test_ship_gates.py, which runs the real wiki arm over a two-file suite with the observer loaded.

Enforcement (amendment)

Mechanical: the cap/refusal/stamp code above + its tests (146 passed, 0 deselected, coordinator re-run) + test-review.py T5 (no os.environ mutation — the gate block that exposed the fabricated-key defect). The original ruling's container-creation gate remains owed per the Mechanism section above; this amendment does not discharge it.

Verification (amendment)

• 2026-10-20: full 4-file cap-test batch green (146/146, 51.90s, -o addopts=""); _child_run_env contract + subprocess integration tests pin the key names against the constants (stamp ⇒ multi-file child green; no stamp ⇒ rc 4, 0 tests).