Knowledge Wiki Orchestrata Ultimate

RUL-027 — Production runs its own checkout, never the development tree

Ruling

Separate dev code from prod (user command, 2026-08-08).

Amended by ADR-0062 (2026-09-23, Proposed — rulings R1/R5): the path clause changes from ~/prod/orchestrata-ultimate to /opt/orchestrata. ~/prod never existed on this host; /opt/orchestrata is a mohamed-owned git checkout reset --hard to the pinned sha by deploy/install-orchestrata.sh, with its own .venv on python3.14t, VERSION promoted last, and an identity verifier that refuses success unless the running units are that checkout. Clauses 2–4 stand unchanged.

Concretely:

  1. Every production unit executes from /opt/orchestrata (was ~/prod/orchestrata-ultimate before ADR-0062), a dedicated git checkout that exists only to be deployed and is reset to a pinned ref by the installer. No WorkingDirectory, ExecStart, EnvironmentFile or ExecStartPre of any prod unit may reference ~/workspace/orchestrata-ultimate-architecture or any /home/*/workspace/ path.
  2. The prod checkout owns its own .venv — a real directory, never a symlink to the development virtualenv. This is RUL-021's rule applied to the deployment target rather than to a feature branch, and for the same reason: a shared environment means one tree's uv sync silently changes another tree's runtime.
  3. Deployment is an explicit checkout of a promoted commit, followed by a restart. Editing files and restarting is not a deployment.
  4. The development tree is never a deployment target. Nothing running there is production, regardless of ORCHESTRATA_ENV_PROFILE.

Why

Before this ruling all six prod units carried WorkingDirectory=/home/mohamed/workspace/orchestrata-ultimate-architecture and Environment=ORCHESTRATA_ENV_PROFILE=prod. Production executed the same files an agent or a human was mid-edit in. Three consequences, all real:

The related 2026-08-05 incident is the same class one layer down: fleet probes silently hit the PROD kernel on localhost:50053 because shells lacked the profile overlay, producing a phantom P1 and two phantom transport bugs and burning hours. Ambiguity about which tree and which endpoint is "prod" has already cost real time twice.

Enforcement

Second mechanism per ADR-0062: scripts/verify_deployment_identity.py (with src/orchestrata_ultimate/promotion/identity.py) proves identity, not health — git -C /opt/orchestrata rev-parse HEAD == VERSION.commit == the expected sha, every orchestrata-*/orchgraph-* unit's process has cwd, cmdline[0] and exe under /opt/orchestrata, and no unit path directive references /home/*/workspace/. It is called by deploy/ship.sh §5, by stages.verify() (FAIL → rollback) and by install-orchestrata.sh --verify. It is listed as prose, not as a gate bullet, until the script lands on master: rulings_lint.py refuses a gate directive whose script does not exist, and that refusal is the point (RUL-014). Promote it to - gate scripts/verify_deployment_identity.py <rule-id> in the commit that adds the script.

Mechanism (RUL-014)

That heading must read exactly ## Enforcement with directive bullets under it: rulings_lint.py parses that heading and no other, so a section named ## Mechanism registers as NO declared mechanism at all. This ruling shipped that way and the lint caught it —

RUL-027: '': no '## Enforcement' section / no bullets found

which is RUL-014's gate working, on the ruling that introduced it.

The mechanism itself is scripts/check_prod_tree_separation.py, wired into scripts/gates.py as the prod-separation gate. It fails when any orchestrata-* user unit references the development tree in a path-bearing directive, and reports the offending unit and line. Prose alone would not have caught the drift that created this ruling.