Isolated test fleet & containerized test services

Since 2026-08-03 the orchestrata suite runs against three physically isolated Incus infra fleets, and the test-env application services (kernel + gateway) run inside a container of the test project — closing a split-brain where the suite's harness talked to prod-pinned host services while pytest targeted a different fleet.

The three fleets

Fleet	Incus project	Subnet	Redis	Graph (Memgraph)	Prolog
prod	`default`	10.53.1.0/24	10.53.1.10	HA cluster, route .160:7691	.16/.17/.18
test	`orchestrata-test`	10.53.2.0/24	10.53.2.11	10.53.2.10:7687 (single)	10.53.2.12
dev	`orchestrata-dev`	10.53.3.0/24	10.53.3.21	10.53.3.20:7687 (single)	10.53.3.22

Isolation is enforced by Incus network ACLs (orch-no-prod/dev/test), not convention: from test you cannot reach prod or dev. dev/test are single infra servers; prod is the only HA fleet.

Endpoint profiles

env/{dev,test,prod}.env are endpoint-only profiles (no secrets — those stay in the gitignored .env). ORCHESTRATA_ENV_PROFILE (default test) selects one. tests/conftest.py hard-pins the suite to env/test.env (overlay with override=True) plus a prod-subnet guard that refuses to run if any resolved infra endpoint sits on 10.53.1.x (override only with ORCHESTRATA_ALLOW_PROD_INFRA=1). Per-worker DB isolation (orchkg_w<N>) is applied in conftest after the overlay, so it is preserved.

Containerized test kernel + gateway

The Agent Runtime kernel and WebSocket gateway for test run as systemd services inside the app container of orchestrata-test — static IP 10.53.2.13 — NOT as host --user services (those target prod). Units + idempotent provisioner: deploy/orchestrata-test/ (provision_app_container.sh). The container needs swi-prolog (the kernel preloads libswipl.so.10); the repo and uv-python toolchain are read-only bind-mounts so the free-threaded .venv runs unchanged.

Kernel gRPC binds 0.0.0.0:50053 → reachable at 10.53.2.13:50053.
Gateway WS/HTTP binds 0.0.0.0:8765/8766 → 10.53.2.13.
Units override *_HOST=0.0.0.0 for the server bind; env/test.env's 10.53.2.13 is the client address.

The split-brain fix

scripts/start_test_services.py used to hardcode localhost in _kernel_config/_gateway_config, so the readiness harness probed/verified the host's prod-pinned kernel+gateway while pytest targeted the test fleet. Now:

_kernel_config/_gateway_config honor the env endpoints; _load_dotenv overlays env/<profile>.env so the whole harness targets the test fleet.
start_kernel/start_gateway refuse to spawn a host process for a remote (containerized) endpoint (_is_local_host) — a host spawn would source .env (prod) and could never bind the container IP.
The gateway readiness uses a dedicated ORCHESTRATA_GATEWAY_PROBE_HOST so pointing the harness at the container never perturbs GATEWAY_HOST, which the gateway server binds (in-process gateway tests must keep binding localhost).

Verify

ORCHESTRATA_ENV_PROFILE=test .venv/bin/python scripts/start_test_services.py --status
# → Redis / Graph / Gateway / Kernel all "running" on 10.53.2.x
ORCHESTRATA_ENV_PROFILE=test .venv/bin/python scripts/run_tests.py
# → 30388 passed / 0 failed / 0 errors, 1507 modules (2026-08-03)