Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0124

ADR-0124: Orchestrata runs in a test container and a prod container built from one OpenMandriva image, over always-on memgraph and Redis

← ADR-0123 · ADR-0125 →

The record

Status: Proposed (item 1 and the image ruled by Mohamed 2026-10-22; the rest proposed) · Date: 2026-10-22

Context

• The host is a poor home for orchestrata's services. On 2026-10-22 ten systemd user units ran out of /opt/orchestrata on a CPU-bound host. Each deploy re-enabled all of them, including units Mohamed had stopped to cut load, and the re-armed drift timer then failed a prod promote (ADR-0123 Context).
• Orchestrata's volks do real work in the workspace. They write specs, code, tests and docs (write_file, run_shell; CodeDirector drives Aider and the Claude CLI), and session.close commits and tags. A read-only checkout mount has already broken it once: the test app container's read-only mount made session.open fail with a PermissionError on .context.lock (wiki project index, 2026-08-05). The kernel's claude_cli provider registers only when a claude binary is on PATH (deploy/orchestrata-test/README.md).
• Precedent: the test fleet already runs the kernel and gateway as systemd units inside the app container (Arch Linux, project orchestrata-test, 10.53.2.13). Measured 2026-10-22: no resource limits, a 6.2 GB root filesystem, 165 MiB RAM in use, the host's uv Python mounted read-only.
• The always-on layer exists: in project default on orch-prod-br (10.53.1.0/24), the orchgraph cluster (coordinators .160/.109/.198, data .128/.123) and redis-queue (.10); the test fleet has its own memgraph, Redis and Prolog on orch-test-br.
• No OpenMandriva image is published: incus image list images: openmandriva returned nothing (2026-10-22). The host runs OpenMandriva Lx 6.0 Rock, kernel 6.14.2.
• The prod deployment at /opt/orchestrata is 3.1 GB.

Decision

1. Two containers (ruled, Mohamed 2026-10-22). A test container, where deployments, runs and services are tested, and a prod container, which is what runs. One is never both.

2. One OpenMandriva image, built by us (ruled, Mohamed 2026-10-22), matching the host's userland. Both containers start from it. It is built when Mohamed says so, not before.

3. Infrastructure stays on, the containers do not (proposed). Memgraph and Redis run 24/7 as infrastructure. Both containers have boot.autostart=false. The prod container joins orch-prod-br and uses the orchgraph cluster and redis-queue. The test container joins orch-test-br and uses the test fleet's memgraph, Redis and Prolog.

4. The workspace is mounted read-write (proposed, from Mohamed: "they need write"), idmapped, so files the volks create are owned by Mohamed on the host. The services run as uid 1001 inside.

5. Services run from a pinned install, never from the mount (proposed). Each container has its own /opt/orchestrata and venv, installed by deploy/install-orchestrata.sh from a pinned ref, with an identity check. A ref reaches prod only after it passes in the test container.

6. Each container carries what the volks use (proposed): git, uv and Python 3.14t, swi-prolog, the claude binary and its credentials, and the provider API keys.

7. Small and limited (proposed): CPU, memory, process and root-disk limits, set from measurement after the first build (starting point 2 CPUs, 2 GiB, about 8 GiB of disk).

8. Claude starts and stops prod, and prod stops itself (proposed). One wrapper command (up, down, status) with an allow rule. up waits for real readiness: the kernel's health check and the gateway's ping, not an open port. The container stops after 15 minutes with no live Claude session lease, no pipeline in flight and an empty conversation queue. Turns wait in Redis while it is off.

9. What moves in (proposed): kernel, gateway, conversation-consumer and ast-live-sync. The orchgraph backup, KG retention and history-mining timers stay on the host, beside the infrastructure they maintain. miniircd and orchestrata-bot follow ADR-0123.

10. The cut-over comes first (ruled, Mohamed 2026-10-22: "cutover first"). The components cut-over (the integration package, INT-02 to INT-25) lands before either container is built. The containers only ever run the components build, never the old single package. Until then the workers build on the host in worktrees, as today.

Open, for later rulings: whether the volks get push credentials (proposed: commit inside, Mohamed pushes); whether QA volks in prod reach the test fleet; the Claude credential method (the refresh-token-stripped mirror or claude setup-token); whether the test container replaces the current app container or stands beside it.

Alternatives rejected

• One container for both testing and running. Mohamed: "we need a test container to test and prod is what is running". Testing in the thing that runs puts every rehearsal on the live system.
• Keep the services on the host. This is the state that cost CPU and that deploys kept re-enabling; nothing bounds it.
• A read-only workspace mount. The volks' work is writing; read-only has already broken session.open.
• A published distribution image (Debian or Arch, as claude-orchestrata and app use). Mohamed asked for an OpenMandriva image matching the host, and none is published.

Consequences

• ADR-0123's "on demand" dispositions for the kernel and gateway become the prod container's lifecycle. Inside a container the units start at container boot; the container is the switch.
• env/prod.env moves the kernel and gateway endpoints to the prod container's address, and the host clients (the orchestrata MCP server, VizInt) connect there. GATEWAY_TOKEN stays a separate fix.
• The host units for the moved services are removed only after the prod container is verified.
• The promoter's prod target becomes the prod container. The pending promoter and installer fix (never enable, try-restart) still applies.
• The image build, the container provisioning and the wrapper each need their own tests and a runtime-view row (ADR-0123).

Provenance

• Mohamed, 2026-10-22, verbatim:
  • "we create an incus container, it will be openmandrive just like this machine with the same kernel because it is a container and we install claude in it and it serves as a test environment, any folders we need in there can just be mounted in, and then we can test the deployments and test the runs and test the services we want and we can reource constrain the container so it doesn't take over the machine, what do you think?"
  • "yes, build our own openmandriva image, but not yet, wait till i finish"
  • "Prod container, services deployed inside run and get the workspace mounted to it and it can be stopped when not needed and should take a small space and resource constrained, when needed, claude can turn it on and off, and when claude is off it turns it off the memgraph and redis remain visible to it and those run 24/7 as infrastructure"
  • "but orchestrata volks and the work it will do, they need write, you need to refresh on what is orchestrata and why we are building it"
  • "no, if you think one container you are wrong, we need a test container to test and prod is what is running"
  • "cutover first" (asked: containers first, running the current package, or the components cut-over first)
• Measurements, 2026-10-22: incus list --all-projects; incus config show app --project orchestrata-test; incus info app; du -sh /opt/orchestrata; incus image list images: openmandriva.

Related ADRs

• Addenda: ADR-0130
• Also mentioned: ADR-0123

Rulings cited

None.

Specs cited

None (or no spec page exists for the ones named).