Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0111

ADR-0111: Test runs never create containers; container creation is exercised only by explicit rehearsals

← ADR-0110 · ADR-0112 →

The record

Status: Accepted (ruling, Mohamed, 2026-10-19, verbatim: "do not test container creation in test runds") · Date: 2026-10-19

Context

On 2026-10-19 the tests that create containers were the largest source of instability on this host.

• Leaks. Project orchestrata held six leaked test instances (ct-prov-*, codetools-ensure-*, codetools-preexisting-*, codetools-bdd-*), the largest 4.1 GiB and running for six days. The cause was teardown that never ran: a timeout kill sends SIGTERM, Python ends without unwinding, and the BDD scenario deleted its instance only after its assertions passed.
• Stalls that broke other test runs. Every test instance lives on t9-btrfs, the same pool as the prod, test and dev fleets, and that pool has btrfs quotas (qgroups) enabled, flagged inconsistent. With quotas on, each instance delete runs quota accounting at transaction commit, and every writer on the filesystem blocks meanwhile.
  • Measured: a test-Redis INCR blocked for 99,256 ms, and in a second sample for 16,922 ms. In the same window a 4 KiB write plus fsync on the T9's ext4 partition took at most 24 ms, so the USB link was not the cause.
  • The bd7aa0dfe ship gate failed on two modules: an IRC session test, because a 300 ms Redis call timed out, and a context-graph test, on Bolt read timeouts against 10.53.2.10.
• Cost. The codetools scratch-provisioner test copies, starts and re-provisions a whole container. It failed four runs on time alone: its in-container rm -rf hit the provisioner's 300 s exec limit.

The leak fix (codetools abd91125) made teardown reliable. It does not remove the churn, the pool-wide stalls or the minutes of I/O per run.

Decision (Mohamed, verbatim: "do not test container creation in test runds")

No test run creates a container or VM. A test run is any test-runner invocation over a project's test tree: a developer run, pytest-live, scripts/run_tests.py, a ship gate, pre-commit, or CI. In such a run, no test may create, copy, launch or init an instance, whether through the incus CLI, the Incus REST API (POST /1.0/instances), a client wrapper (create_instance, ensure_instance, copy_instance, launch) or terraform. Tests may read and exec into the standing fleet containers that already exist.

Code whose job is creating containers is exercised by explicit rehearsals instead:

• Examples are codetools' Incus client and gate provisioner, and orchestrata's provisioners.
• A rehearsal lives outside the test tree, so no test run collects it.
• It is run deliberately, by a person or an orchestrator, when that code changes.
• It records its evidence the way a deployment rehearsal does.

Each repository's test run carries a policy gate that fails when a module under its test tree calls an instance-creating API. Prevention is static, so it does not depend on remembering the rule.

Alternatives rejected

• Keep the container tests, marked live-only, and run them when the Incus code changes. This was the orchestrator's recommendation the same day. Ruled out: it is still a test run creating containers.
• Keep them in every run, and fix only the pool (a quota-free pool, or a separate scratch pool). This removes the stall but keeps the churn, the leaks and the minutes of I/O in every gate.
• Replace them with a scripted fake Incus server. The no-mocks rule (RUL-002) forbids it, and a fake proves nothing about the real daemon. The codetools unit tests that already drive a scripted HTTP server for protocol edges are unaffected; they create nothing.

Consequences

• codetools:
  • These leave the test tree for rehearsals/:
    • tests/gate/test_provisioner_scratch_container.py;
    • tests/incus/live/test_idempotent_ensure.py;
    • the instance-creating scenarios of tests/bdd/test_spec_013.py;
    • the live case of tests/test_incus_isolation.py.
  • The abd91125 isolation helper (owned names, owner-pid tags, orphan reclaim, signal-safe finalizers) moves with them, since rehearsals still create instances.
  • SPEC-013's creation requirements lose automated test-run evidence. Their RTM grade drops until a rehearsal records evidence for them. That is stated, not hidden.
• orchestrata:
  • The incus_live lane leaves the test tree for rehearsals/.
  • So does the INCUS_HARNESS_IT=1 round-trip in tests/harness/test_incus_isolation.py, which already failed every default run because it raises when the variable is unset.
  • So does any other module the census finds creating instances.
• Both repos gain the policy gate in their test run. A new instance-creating test fails the run it is added to.
• Separate decision: whether t9-btrfs keeps btrfs quotas is not settled here. With the test churn gone, the pool's remaining deletes come from deliberate operations.

Provenance

• Mohamed, 2026-10-19, asked "why are you creating containers during test run?". The orchestrator listed the creators: three codetools test modules, a probe written by the leak-fix builder to prove signal-safe teardown, and orchestrata's env-gated harness round-trip.
• He then ruled, verbatim: "do not test container creation in test runds".
• Evidence:
  • the codetools leak-fix report plans/briefs/2026-10-19-test-instance-leaks-report.md (abd91125);
  • the Redis latency probes and the T9 kernel log, both 2026-10-19;
  • the failed bd7aa0dfe ship gate (reports/run-tests-latest.json in wt-CONV-GUARD).
• Related: RUL-036 (tests categorised functional or integration by dependency), RUL-002 (no mocks).

Related ADRs

No other ADR is named in this record.

Rulings cited

• RUL-002: RUL-002 — No mocking — real infra, real implementations only
• RUL-036: RUL-036 — Every test is categorised functional or integration, by what it depends on

Specs cited

• SPEC-013