Orchestrata Ultimate › Plans › prod-deployment-isolation-design

Prod deployment isolation — design

Source: plans/prod-deployment-isolation-design.md at 8a2cb42ad · last changed 2026-08-07 · first committed 2026-08-07 · 196 lines.

Prose pending.

Plan text

Prod deployment isolation — design

Investigated 2026-08-07, read-only, in /home/mohamed/workspace/orchestrata-ultimate-architecture, branch feat/cxp-direct-transport, currently dirty (git status --short: 20+ modified/deleted files, none committed, including src/orchestrata_ultimate/agents/runtime_main.py) — live evidence for (b): every prod unit below is already carrying these uncommitted edits.

(a) Every prod unit today — facts

7 systemctl --user units. Declared source is deploy/systemd/ (deploy/systemd/README.md:9-11); scripts/health_check.py:465-502 (check_systemd_units, wired at scripts/health_check.py:559,573) asserts the installed copies match — confirmed true here via systemctl --user cat.

Every row's WorkingDirectory=/EnvironmentFile=.env/EnvironmentFile=-env/prod.env is the literal string /home/mohamed/workspace/orchestrata-ultimate-architecture — the primary entry of git worktree list (… 186a3690 [feat/cxp-direct-transport]), the tree this investigation ran in. The 5 daemon units also run ExecStartPre=/usr/bin/timeout 180 /home/mohamed/.local/bin/orchestrata-kg-ready (e.g. deploy/systemd/orchestrata-kernel.service:30); that probe is not in this repo (grep -rn orchestrata-kg-ready . hits only context.md:587 prose) and its shebang is #!.../orchestrata-ultimate-architecture/.venv/bin/python — dev-tree dependent too.

CXP_GRPC_HOST=localhost at env/prod.env:34 is load-bearing (kernel wildcard-binds 0.0.0.0:50053 on the host — per ~/.claude/…/memory/project-env-prod-base-layer.md). .env is gitignored (.gitignore:105; git ls-files .env → empty): a per-host secret, not code, correctly outside what promotion should touch. .venv/bin/python is a symlink to the shared interpreter binary at ~/.local/share/uv/python/…/python3.14t (fine), but .venv itself sits at the dev-tree root and is 6.9 GB (du -sh .venv) of deps + native extensions built against whatever source was present at uv sync time.

(b) Failure modes — concrete trigger sequences

1. Silent branch-flip on next restart. Dev runs git checkout master in the dev tree. No prod process reacts immediately. The next systemctl --user restart orchestrata-kernel.service — manual, or automatic via Restart=on-failure after any crash (deploy/systemd/orchestrata-kernel.service:33) — re-execs against whatever is checked out at that instant. Nothing records or checks which commit prod is meant to run.
2. Uncommitted edit is already live. No git operation needed at all: an edit under WorkingDirectory= is already what the next restart boots. Verified today — runtime_main.py is modified, uncommitted, ships on the next kernel restart.
3. A broken checkout crash-loops all 5 daemons together. They share one WorkingDirectory/.venv; a mid-rebase or bad-merge state that breaks import orchestrata_ultimate trips Restart=on-failure + StartLimitBurst=5 within StartLimitIntervalSec=300 (deploy/systemd/orchestrata-kernel.service:9-10) for all 5 at once.
4. uv sync mid-flight breaks every unit's readiness probe. All 5 daemons gate startup on orchestrata-kg-ready, whose shebang is the dev tree's .venv/bin/python. A dev-side uv sync --dev rebuilding janus-swi/cxpt/ctok (pyproject.toml:372,378,384) leaves that interpreter transiently broken; any unit needing to (re)start in that window fails ExecStartPre before reaching ExecStart.
5. Dev tooling mutates the tree prod execs from. uv add, git clean, or a generator (scripts/gen_cxp.py, scripts/serialize_codebase_ast.py — both currently modified per git status) rewrite files under the same tree prod runs from. No isolation between "iterating on this branch" and "what prod runs."

(c) Design: isolated prod checkout + venv

Location. git worktree add /home/mohamed/workspace/orchestrata-prod-release prod-release — a worktree, not a second clone: shares the 164 MB .git object store (du -sh .git), enumerable by the same git worktree list --porcelain call scripts/check_worktree_envs.py:112-129 and scripts/worktrees.py already use. prod-release is a branch reserved for this purpose alone — never checked out in the primary tree or any wt-* worktree, so an interactive git checkout prod-release never happens by accident.

Per RUL-021 (scripts/check_worktree_envs.py:1-10, the wt-QWEN incident) this worktree gets its own non-symlinked .venv, built with uv sync --dev --frozen from its own pyproject.toml/uv.lock (tracked: git ls-files uv.lock). Every native dependency is git-tracked source, not a binary the dev tree happens to have built: scripts/_janus_patched (50 files), scripts/_cxpt (6), scripts/_ctok (21), vendor/wheels/*.whl (5, un-ignored at .gitignore:45-46). --frozen refuses to touch uv.lock, so promotion never silently upgrades a dep the test-env run never saw.

Update procedure. scripts/promote_env.py's Promoter.promote() (scripts/promote_env.py:316-327) already gates on a GREEN test-env run (gate_for, scripts/promote_env.py:224-244) but only appends a ledger record — no git, no venv, no restart. Extend it (deploy --to prod) to, inside $PROD_CHECKOUT:

1. Resolve <ref> (default origin/master); refuse unless git merge-base --is-ancestor <ref> origin/master. Re-run gate_for("prod") unchanged.
2. git -C $PROD_CHECKOUT fetch origin && git -C $PROD_CHECKOUT checkout --detach <ref> — the only place this ref is ever materialized as files.
3. uv sync --dev --frozen in $PROD_CHECKOUT — rebuild before restart, with a clean build/ for any changed vendored native package (the uv-serves-stale-native-wheels staleness class this project already tracks).
4. scripts/check_source_provenance.py --python $PROD_CHECKOUT/.venv/bin/python — the gate already supports an explicit --python (scripts/check_source_provenance.py:90-95); proves the fresh venv imports from $PROD_CHECKOUT/src.
5. systemctl --user restart the 4 daemon units — safe now because WorkingDirectory=$PROD_CHECKOUT (below), so the restart boots the commit from step 2, never whatever the primary tree has checked out.
6. Record the SHA in reports/promotion-state.json via the existing Promoter.record() (scripts/promote_env.py:218-223).

Unit files. All 7 files in deploy/systemd/ get every /home/mohamed/workspace/orchestrata-ultimate-architecture path (WorkingDirectory=, both EnvironmentFile=, every ExecStart=/ ExecStartPre=) rewritten to $PROD_CHECKOUT, then deploy/systemd/install.sh re-run (already diffs + reinstalls only changed units, deploy/systemd/install.sh:20-24). .env is copied into $PROD_CHECKOUT once, out of band, staying gitignored — promotion never touches it.

Readiness probe. Move ~/.local/bin/orchestrata-kg-ready into the repo (deploy/systemd/orchestrata-kg-ready.py) so it ships with the promoted commit and its shebang resolves against $PROD_CHECKOUT/.venv, instead of staying a hand-authored, untracked file hardcoding the dev tree.

(d) Design: a gate that fails on dev-tree WorkingDirectory

Enumeration. Read deploy/systemd/*.service directly — static files, no live systemctl dependency, matching every GATES entry in scripts/gates.py (content-hashed input roots, scripts/gates.py:90-110). Extract WorkingDirectory= with a line-oriented parse, style of scripts/check_worktree_envs.py:88-109.

"Is a dev tree" test. Resolve each WorkingDirectory and compare it against every path git worktree list --porcelain reports — reusing scripts/check_worktree_envs.py:112-129's list_worktrees by import, not reimplementing it. Violation = the path matches any registered worktree (primary or any wt-*), full stop. This beats the alternatives: a branch-name check is defeatable by renaming a branch and false-PASSes the instant someone runs git checkout prod-release in the primary tree by mistake — the exact accident this gate exists to catch; a dirty-tree check false-PASSes a clean tree one git checkout away from disaster — (b.1) starts from a clean switch, not a dirty one. Worktree-registry membership (chosen) is structural — independent of current branch or dirty state — and also catches a "fix" that repoints prod at a wt-* feature worktree instead of the primary.

Secondary violation: the 7 units' WorkingDirectory values must all be identical — a partial migration (kernel fixed, orchgraph-backup still on the dev tree) is exactly how the .env-overlay rollout happened unit-by-unit here (deploy/systemd/orchestrata-kernel.service:20-22; context.md:587: 5 of 7 units drifted behind the other 2 on the readiness-probe fix).

Exit-code contract, matching scripts/check_worktree_envs.py:379-396's skip-if-not-applicable rule: exit 0 SKIP if deploy/systemd/ is absent or has zero *.service files; exit 0 OK if every WorkingDirectory resolves outside the worktree registry and all match; exit 1 FAIL, one line per offending unit, same [gate] STATUS detail format as scripts/check_worktree_envs.py:399-419.

Registration in scripts/gates.py GATES (scripts/gates.py:59-87): "prod-tree-isolation": ([sys.executable, "scripts/check_prod_tree_isolation.py"], ("deploy/systemd",)).

Companion registration. scripts/gates.py's GATES is the RUL-008 review battery; the two gates named in the task brief are not actually in it — check_worktree_envs.py is wired into scripts/health_check.py's build_checks() (scripts/health_check.py:559) and check_source_provenance.py into scripts/run_tests.py:230's preflight, two different registries. For this gate to run where its siblings do, also register it in build_checks() (scripts/health_check.py:549-568), right after ("Systemd units (deploy/systemd)", check_systemd_units) (scripts/health_check.py:573), same wrapper shape as check_worktree_envs (scripts/health_check.py:129-137).

(e) Risks and mitigations

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks