Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0062

ADR-0062: Production runs from a pinned-ref git checkout under /opt/orchestrata

← ADR-0061 · ADR-0063 →

The record

Date: 2026-09-23 · Status: Proposed (awaiting Mohamed's ruling)

Context

RUL-027 (2026-08-08) moved production off the development tree and named ~/prod/orchestrata-ultimate as the dedicated checkout every prod unit executes from. Measured read-only on 2026-09-23 (plan baseline): ~/prod does not exist on this host and never ran anything here, and /opt/orchestrata is absent on the host as well. ADR-0053 records Mohamed's deliberate deletion of the prod stack and the law that follows from it — infrastructure claims carry live probe evidence or are violations, and a document asserting dead infrastructure is a defect. A ruling whose path clause names a directory that does not exist is that defect.

Meanwhile the test container app (10.53.2.13, project orchestrata-test) already runs a git-archive copy of the tree at /opt/orchestrata (VERSION 3cf440b39, architecture branch) under SYSTEM units, so host and container disagree about what the production root is. Master's provision_app_container.sh is still the bind-mount version hard-coding the -architecture path.

The deployment methodology (claude/workflow/deployment.md) states the five properties without which a service is not deployed:

1. copies under /opt/<name>, outside every working tree;
2. its own uv venv on python3.14t (/opt/<name>/.venv);
3. the source commit recorded in the deployed tree;
4. an installer that re-deploys from a pinned git ref — git reset --hard <ref>, never a merge;
5. a verification step that REFUSES success unless the running thing is the deployed thing.

The promotion package (src/orchestrata_ultimate/promotion/{runner,stages, models,projects}.py) already implements the tree/drift/rollback stages, and its advance_tree, drift and _rollback need .git in the target.

Decision (proposed)

Rulings R1 and R5 of the 2026-09-23 plan, recorded as one decision:

• /opt/orchestrata is a mohamed-owned git checkout, reset --hard to the pinned sha by deploy/install-orchestrata.sh (the dr-hannah reference-installer pattern). Never a merge, so the tree cannot accumulate local state; rollback is a reset to the previous sha, not a re-install.
• It owns its .venv on python3.14t (uv sync --frozen --python).
• VERSION is staged as .VERSION.deploying and promoted last — only after the identity verifier passes.
• .env is copied in (gitignored, so it survives resets) and installed 0600.
• The identity verifier (src/orchestrata_ultimate/promotion/identity.py + scripts/verify_deployment_identity.py) refuses success unless the running units are that checkout: git -C /opt/orchestrata rev-parse HEAD == VERSION.commit == --expect-sha; every discovered orchestrata-* / orchgraph-* unit's MainPID has cwd == /opt/orchestrata, cmdline[0] under it and exe resolving to the target of /opt/orchestrata/.venv/bin/ python; process start time ≥ VERSION.deployed; no path directive in any discovered unit references /home/*/workspace/; zero in-scope units is exit 2, never success.
• Units stay user scope this round (what runs today; Linger on). Scope is resolved per unit in code (systemctl --user cat <unit> succeeds → user, else system), so a later move to system units is a unit-file change only.
• This supersedes RUL-027's path clause only (~/prod/orchestrata-ultimate → /opt/orchestrata). Its other clauses — the prod checkout owns its own .venv, deployment is an explicit checkout of a promoted commit, the development tree is never a target — stand unchanged and gain the verifier as a second mechanism.

Alternatives rejected

1. Keep ~/prod/orchestrata-ultimate as the git checkout (RUL-027 as written). Rejected: three shapes across dev/test/prod (workspace tree, container archive at /opt/orchestrata, ~/prod checkout), and a mutable HEAD in a home directory as the only record of what is deployed. It also does not exist on this host, which ADR-0053 makes a violation to keep asserting.
2. A git-archive copy at /opt/orchestrata (the install-tools.sh pattern the test container runs; R1's first phrasing). Rejected for prod by R5: promote_env's tree/drift/rollback stages need .git, so an archive copy forks the promotion path and turns rollback into a re-install. The container keeps its archive copy — same layout, the one documented divergence — because nothing promotes into it.

Consequences

• RUL-027 is amended in the wiki, not retired: path clause → /opt/orchestrata, ADR-0062 named as the amending decision, scripts/verify_deployment_identity.py named under its ## Enforcement (it becomes a gate directive there the commit the script lands in the tree; rulings_lint.py refuses a gate whose script does not exist).
• scripts/check_prod_tree_separation.py's message names /opt/orchestrata as the expected root; its globs gain orchgraph-*.service and *.timer.
• deploy/ship.sh §5 calls scripts/verify_deployment_identity.py (replacing steps 5a+5b); stages.verify() calls it after its poll (FAIL → rollback); install-orchestrata.sh --verify calls it.
• All eight prod units point at /opt/orchestrata; orchestrata-kg-ready moves into the tree as deploy/orchestrata-kg-ready.py.
• Follow-up, not this round: system units with User=mohamed (deployment.md's rule for services that must always run) is a unit-file change once scope is resolved per unit.
• Related rulings from the same session, recorded here rather than as separate ADRs:
  • R3 — deploy assets are cherry-picked, never merged. deploy.sh, the provisioners, the ast/codemod units and ADR-0046/0047/0048 come from architecture/domain-model onto master by cherry-pick in a worktree off origin/master; no full merge of that branch.
  • R4 — the rehearsal HA cluster lives on the test subnet. The rh-coord-1/2/3 cluster re-IPs onto 10.53.2.x, TEST_MEMGRAPH_COORDINATOR_URIS follows, and Bolt is verified on the new addresses (the plan implements this by rebuilding the cluster inside project orchestrata-test, because Raft state embeds --coordinator-hostname).

Provenance

Ruled by Mohamed on 2026-09-23 in the codetools planning session: rulings R1 and R5 (and R3, R4 above) in /home/mohamed/workspace/orchestrata.codetools/plans/PLAN-2026-09-23-production-deployable-tools.md, sections "Rulings from Mohamed" and "Design: orchestrata-ultimate production path". Plan mode forbade commits, so this file is a Round-0 task and the ruling text in the plan is authoritative until this record is Accepted. Baseline facts (~/prod and /opt/orchestrata absent on the host; container app running an archive copy under system units; next free id 0062 because 0059–0061 exist on the architecture branch) were measured read-only on 2026-09-23. Status stays Proposed until Mohamed flips it.

Related ADRs

• Also mentioned: ADR-0053, ADR-0046

Rulings cited

• RUL-027: RUL-027 — Production runs its own checkout, never the development tree

Specs cited

None (or no spec page exists for the ones named).