Orchestrata Ultimate / Deployment

Deployment

This page is the hub for how the platform reaches the machine. It covers seven deployments:

1. the product at /opt/orchestrata;
2. the observability tools at /opt/orchestrata-tools;
3. codetools at /opt/codetools;
4. hermes-subagent at /opt/hermes;
5. pi-task at /opt/pi-worker;
6. the user-level Claude configuration in ~/.claude, with the orchestrata plugin;
7. the Incus fleets, owned by OpenTofu.

Every statement below comes from the installer and launcher sources, the deploy logs, or read-only commands run on 2026-10-22 and 2026-10-23. Sources:

• installers: deploy/install-orchestrata.sh, deploy/install-claude-config.sh, deploy/install-tools.sh and deploy/ship.sh in the worktree wt-MAP-PRECOMMIT at 8a2cb42ad;
• ~/workspace/hermes/deploy/install.sh and ~/workspace/pi-harness/artifacts/pi-worker/install.sh;
• launchers: ~/bin/ct-install, ~/bin/ct-smoke, ~/bin/tools-deploy and ~/bin/hermes-deploy.

Nothing on this page was deployed, started or stopped to write it.

Companion pages:

• ship and deploy: what ship does, step by step.
• Deployment sequence: every script, its order, and where each component lands.
• Deployment inventory: the measured state, item by item.
• Host services: what each deployed unit does.
• codetools deploy: the codetools release engine in detail.

Platform conventions

Six rules shape every deployment. Each one is in the installers as a check, not only as prose.

There is one deliberate exception to the first rule. The Hermes Agent venv inside a Hermes release is built on the Python version that deploy/hermes-agent.lock pins. The installer header says why: "Hermes requires < 3.14". The release VERSION records agent_python 3.13.5.

Deployment architecture

The plugin path is the one indirect delivery. install-claude-config.sh installs the orchestrata plugin with claude plugin install from the git remote, so the plugin can only ever be a pushed commit.

Current deployments (measured 2026-10-22/23)

The Incus fleets have no single commit; see Incus fleets and the infrastructure hub.

/opt/orchestrata - the product

What it is. A release checkout of orchestrata-ultimate-architecture, owned by the deploying user, reset to a pinned sha. Every orchestrata user unit runs /opt/orchestrata/.venv/bin/python with WorkingDirectory=/opt/orchestrata (ADR-0062).

Installer. deploy/install-orchestrata.sh [--ref REF] [--prefix DIR] [--bootstrap-only] [--verify] [--python PATH] [--secrets-env FILE]. The default ref is origin/master.

What it does, in order:

1. It fetches origin in the source checkout and resolves the ref to a sha. It refuses the sha unless it is an ancestor of origin/master.
2. It records the previous HEAD and the previous VERSION text, for rollback.
3. It clones the origin URL into the prefix if needed, then runs git reset --hard <sha>. The script comments on the choice: "reset --hard, not merge: production must equal the pinned sha".
4. It creates .venv on ~/.local/bin/python3.14t if absent. It loads the compiler from env/toolchain.env through deploy/toolchain.sh, then runs uv sync --frozen --no-editable.
5. It runs scripts/worktree_toolchain.py from the prefix's own venv. This proves every locked path package imports, because uv sync can exit 0 while a native package failed to build.
6. It asserts the deployed venv is free-threaded.
7. It installs the secrets file as /opt/orchestrata/.env with mode 0600.
   • The source is ~/.config/orchestrata/.env (ADR-0066, ruling R16). It is never the workspace template.
   • A missing file is refused, and so is a file whose keys still hold CHANGEME placeholders. Only the key names are reported.
8. It writes VERSION (commit, ref, repo, deployed) before any restart, so every later reader sees the new record.
9. Unless --bootstrap-only is set, it restarts every orchestrata-*.service and orchgraph-*.service in both the user and the system manager. Three kinds of unit are skipped:
   • orchestrata-canvas.service, under ruling R8 (it is not an /opt/orchestrata unit);
   • timer-driven oneshots;
   • units whose Requires= or BindsTo= names a unit this host cannot load.
10. With --verify, it runs scripts/verify_deployment_identity.py --root PREFIX --expect-sha SHA from the prefix's venv. When pyproject.toml declares orchestrata-mcp, it also checks that .venv/bin/orchestrata-mcp resolves inside the prefix.

Rollback. If verification fails, the installer:

• writes the previous VERSION text back, or removes VERSION on a first install;
• runs git reset --hard to the previous sha;
• restarts the units again, unless the run was --bootstrap-only.

Promotion. Advancing prod is a promotion, not a side effect of shipping. scripts/promote_env.py promote --to prod runs the stages in promotion/runner.py::deploy, in this order: resolve, infra, tree, deps, assets, record_version, services and verify. Then graphs runs unless graphs are skipped, and ingest runs unless --skip-ingest is set. A failed verify rolls back (report.rolled_back = _rollback(...)).

The module docstring of promotion/stages.py explains the order: "The tree moves before dependencies sync ... Assets install before services restart ... Verification runs after the restart and before the graphs".

ship runs promote_env.py test test first. That run records the GREEN test-env result the prod gate reads. Then ship runs promote --to prod --yes --skip-ingest and checks two things:

• /opt/orchestrata HEAD equals origin/master (check 5a);
• every unit whose ExecStart is under /opt/orchestrata/ is active, and every timer-driven oneshot has an active timer (check 5b).

Last deploy. On 2026-10-22 at 15:35, install-orchestrata.sh --bootstrap-only re-installed prod clean at 4ac0db570, and no unit was touched. See host services.

/opt/orchestrata-tools - the project-agnostic tools

What it is. The tools are ship, swarm_supervisor, observe_pytest, pytest-live, ast_live_sync, ast-daemon, codemod-daemon and ast, with a vendored lib/ subtree of orchestrata_ultimate and orchestrata_ast. They have their own .venv and a one-module shim/. That shim is the pytest observer, which ~/.claude/settings.json puts on PYTHONPATH. /usr/local/bin links ship, swarm-supervisor, observe-pytest, pytest-live and ast-sync into bin/.

Launcher. tools-deploy [--check] COMMIT in ~/bin, approved by Mohamed on 2026-10-22, with allow rules in ~/.claude/settings.local.json. It:

1. refuses unless the commit exists in ~/workspace/orchestrata-ultimate;
2. takes a flock on ~/.local/state/tools-deploy/lock;
3. checks the commit out into its own sparse, detached worktree ~/.local/state/tools-deploy/tree, which holds only deploy/ and env/;
4. refuses local changes in that tree, a HEAD that is not the commit, a deploy/ or env/ that differs from the commit, and an installer that bash -n cannot parse;
5. runs that commit's deploy/install-tools.sh --ref COMMIT;
6. fails unless /opt/orchestrata-tools/VERSION records the commit.

The log is ~/.local/state/tools-deploy/<commit12>.log, and its last line is DEPLOY-EXIT <code>. --check stops before the installer.

Installer. deploy/install-tools.sh [--ref SHA] [--prefix DIR].

1. It reads deploy/tools-vendor-manifest.txt at the sha. That file is the computed import closure of the deployed entry modules.
2. It runs git archive over the listed scripts, the vendored src/ subtrees and deploy/{ship.sh,retire-plain-units.sh,restart-tool-instances.sh,systemd}.
3. It removes lib/, bin/ and shim/ and recreates them.
4. It installs the measured dependencies. It runs tools_import_closure.py --check, which fails the install if the staged tree cannot resolve its own imports.
5. It builds the native codemod_fast and ctok extensions with --no-build-isolation and the compiler from env/toolchain.env.
6. It writes the entry points and stages .VERSION.deploying.

Then it verifies. A 2026-10-22 run of 2c29cd9d3 logged 23 checks, among them:

• every entry point resolves under the prefix and none into a checkout;
• pytest-live carries the shim, not lib/;
• the venv is free-threaded;
• ship on PATH is the deployed artifact;
• an ast-daemon started from the deployed venv answers /health with deployed_commit equal to the sha, and answers a symbols query;
• codemod-daemon reports the same identity with codemod_fast loaded;
• both daemons exit on SIGTERM.

It then installs the orchestrata-ast-daemon@ and orchestrata-codemod-daemon@ templates, checks them byte-identical to the staged copies with systemd-analyze verify, and retires the plain checkout-bound units. Only then does it promote VERSION and restart every active instance of the two templates.

Rollback. There is no automatic rollback of the code. On failure, the exit trap restores the previous VERSION (or removes it on a first install), but bin/, lib/ and shim/ were already replaced at step 3. To roll back, deploy the previous commit with tools-deploy PREVIOUS. The scratch log tools_rollback.log of 2026-10-22 is exactly that: an install of 44265c096.

/opt/codetools - the codetools release engine

What it is. codetools installs itself with its own deploy activity. Each release is a directory /opt/codetools/releases/<release-hash>/ with its own .venv. current is a symlink to one release. bin, .venv and VERSION at the prefix root are symlinks through current. On 2026-10-23, current pointed at releases/58dc55ce7d9ba893 and VERSION recorded commit a5c419f5720972267039adaf5efc932b19f835a4. Details: codetools deploy.

Launcher. ct-install LABEL [REF] in ~/bin. The default ref is HEAD of ~/workspace/orchestrata.codetools. Since 2026-10-22 it follows the Hermes model (Mohamed: "make the installer like hermes"). It:

1. unsets PYTHONPATH and PYTEST_ADDOPTS, because install 68 died on the inherited orchestrata-tools shim;
2. refuses when an install is already running, when reports/ct8-install<label>.log already exists, or when the tree is dirty. Its own logs and .claude/ do not count as dirty;
3. runs detached under systemd-run --user --scope with PYINFRA_VENV, CC=/usr/bin/gcc and CXX=/usr/bin/g++.

Inside that scope:

1. uv run codetools deploy install --repo . --ref SHA --prefix /opt/codetools --python python3.14t --no-verify builds the release from the lockfile and promotes it with no test-suite gate. The tests run before the merge.
2. /opt/codetools/bin/codetools deploy verify-identity --prefix /opt/codetools --expect-commit SHA proves the identity through the installed command.
3. ~/bin/ct-smoke SHA runs live calls. --version must name the commit; ast map must map a three-symbol fixture repo and render class Child(Base); mcp list must carry the ast map tool.
4. If either check fails, codetools deploy rollback --prefix /opt/codetools runs and the exit code becomes 12.

The log ends with WALL <s> and INSTALL-EXIT <code>.

Measured. Install 73 (e1ffbc3) took 26 s, according to the launcher's memory note. Install 75 (a5c419f5) logged WALL 43s. The suite-gated verify is still available by running codetools deploy install without --no-verify. It took minutes: install 72 spent 60 s selecting --affected-scope and then more than 3 min in 8 shards.

Rollback. codetools deploy rollback --prefix P re-points current at the previous release. That is possible because previous release directories are still on disk: 52 were measured. ADR-0035 (Accepted) says exactly one release is kept, and rollback is a reinstall from a tag. The code and the ADR disagree; see codetools blockers.

MCP sessions keep their release. A Claude session started before a promote keeps the codetools mcp serve process it launched. On 2026-10-22 at 23:30, three server processes were still running /opt/codetools/releases/6b4babfeaac2e008/.venv/bin/python3.

/opt/hermes - hermes-subagent

What it is. Hermes Agent (Nous Research) wrapped as a Claude Code subagent and the user-scope MCP server hermes. Releases are immutable directories /opt/hermes/releases/<commit>/, and current is a relative symlink. The live release, b1b66a6, pins Hermes Agent 9244ec1d774c from https://github.com/NousResearch/hermes-agent.git.

Launcher. hermes-deploy COMMIT in ~/bin, approved by Mohamed on 2026-10-22, with an allow rule in settings.local.json. It refuses unless:

• the commit exists in ~/workspace/hermes;
• the repo is on master and clean;
• master can fast-forward to the commit;
• the lock is free;
• DEEPSEEK_API_KEY is set in ~/.env.

It exports only DEEPSEEK_API_KEY and OPENROUTER_API_KEY from ~/.env, and prints no value. Then it runs git merge --ff-only COMMIT followed by deploy/install.sh COMMIT. The log is ~/.local/state/hermes-deploy/<commit12>.log, ending in DEPLOY-EXIT <code>.

Installer. deploy/install.sh [REF]:

1. It refuses an existing finished release ("releases are immutable") and removes an unfinished one.
2. It runs git archive of the commit into releases/<commit> and writes MANIFEST, the sha256 of every extracted file.
3. It fetches the pinned Hermes Agent commit into a bare cache, refuses any other commit, extracts it to agent/ and writes AGENT_MANIFEST.
4. It builds both venvs with uv sync --frozen --no-dev: the release on python3.14t (GIL off, checked), and the agent on the lock's python.
5. It writes VERSION and runs hermes_subagent.cli verify --release-only. A failure here leaves current unchanged.
6. It wires the home directory and copies the Claude skill and agent definition from the release.
7. It switches current atomically (ln -sfn to current.new, then mv -T).
8. It writes bin/{hermes-task,hermes-mcp,hermes-guard} and re-registers the hermes MCP server at user scope.
9. It runs hermes-task verify --live through the installed command. That check covers the identity of every deployed piece and one live PONG per provider with a key. Only a failure of the default provider refuses.
10. It removes replaced releases, except one still locked by a running task.

Rollback. Any failure after current moved re-wires the previous release, points current back at it and removes the new release. On a first install, it unregisters the MCP server and removes the skill and agent definition. A recorded rollback from 2026-10-22, in the session's scratch notes: d959cb9 built and verified, then failed the live DeepSeek call with HTTP 402 Insufficient Balance and was rolled back to 3549d30. After a top-up, the redeploy succeeded. A rollback leaves master at the new commit.

/opt/pi-worker - pi-task

What it is. The deployed CLI pi-task for isolated Pi workers in the Incus project pi-workers, with a container acceptance judge. /opt/pi-worker/current points at releases/238d899c3300..., and bin/pi-task is linked from ~/bin/pi-task.

Installer. ~/workspace/pi-harness/artifacts/pi-worker/install.sh [REF]. There is no ~/bin launcher. It refuses unless:

• the ref is a commit;
• the release does not exist yet;
• the live image's fingerprint equals the one image-record.env records at that commit;
• the image was built from the same pi-goal and pi-guard trees.

It extracts into releases/.incoming-<commit>-<pid> and builds the venv on python3.14t (GIL off, checked). It writes VERSION with commit, path, ref, repo, deployed, tree, image and python, then moves the directory into place.

Verification. pi-task verify runs from the new release before current moves, and again through the installed wrapper after. The project notes say it refuses unless the running release is at its recorded path, matches its recorded file tree, runs on its own python3.14t venv, and drives the recorded image.

Rollback. A failed first verify leaves current unchanged. The second verify, after the switch, has no rollback step in the script: under set -e it exits non-zero. The replaced releases are still on disk at that point, because removal runs only after that verify passes.

~/.claude and the orchestrata plugin

What it is. The user-level Claude Code configuration, developed in the repo's claude/ tree. It covers the subtrees hooks, tools, agents, skills, commands and workflow, plus statusline-command.sh, settings.hooks.json, .envrc, scripts/patch_settings.py and scripts/register_hooks.py. The orchestrata plugin is the only hook delivery path. Runtime state such as projects/, history.jsonl and caches is not touched.

Installer. deploy/install-claude-config.sh [--ref SHA] [--prefix DIR] [--verify] [--mcp-only]. It:

1. requires the claude CLI;
2. outside --verify, refuses a sha that is not on origin/master as last fetched: "the plugin delivers pushed commits only";
3. runs git archive SHA claude into a temp stage and refuses symlinks;
4. creates ~/.claude/.venv on python3.14t and installs the dependencies compiled from claude/pyproject.toml (the fallback set is jinja2 neo4j pydantic redis);
5. copies each existing managed subtree to ~/.claude/.backup-<timestamp>/ and keeps the 3 newest backups;
6. runs rsync -a --delete from each staged subtree into ~/.claude/<subtree>/. That is every file, with no filename heuristics;
7. rewrites Python shebangs in hooks/ and tools/ to the deployed venv;
8. copies the loose files and stages .VERSION.deploying;
9. retargets settings.json and runs direnv allow;
10. registers the MCP servers at user scope through claude mcp remove and claude mcp add-json:
    • codetools from the fragment codetools deploy render-mcp writes, /opt/codetools/bin/codetools mcp serve;
    • orchestrata as /opt/orchestrata/.venv/bin/orchestrata-mcp.

The plugin re-pin. The marketplace must be the git remote. A directory or other source is uninstalled and re-added. If the installed gitCommitSha is not the deploy sha, the installer tries claude plugin update, then uninstalls and reinstalls. claude plugin update compares version strings only, and the plugin cache is keyed by version (cache/orchestrata/orchestrata/1.5.0). So same-version content changes never update, and uninstall plus install is the only supported way to move them. The script notes that claude 2.1.236 has no --force or --refresh flag.

Verification. It runs in both modes; --verify skips every mutation, so a verifier cannot repair what it measures. It checks:

• the plugin is enabled, and its recorded gitCommitSha equals the sha;
• every plugin file in the live cache is byte-identical to the commit (222 on 2026-10-23);
• every deployed file is byte-identical from line 2 on (143 on 2026-10-23), and line 1 is either unchanged or a shebang retargeted to the venv;
• every shebang points at the deployed venv;
• there are no symlinks;
• the venv is free-threaded and the hook dependencies import;
• settings.json targets the venv;
• every declared hook is wired (register_hooks.py --check --plugin auto);
• a real hook, platform_conventions_prompt.py, runs;
• no deployed file references the source checkout;
• both registered MCP commands resolve inside their releases (codetools deploy verify-identity --mcp-json for codetools, a path check for orchestrata).

VERSION is promoted from the staging file only after all of that passes.

Rollback. There is no automatic rollback. A failure leaves .VERSION.deploying behind, VERSION still names the previous commit, and --verify then fails loudly on the half-copied tree. The backups under ~/.claude/.backup-* are the manual rollback ledge.

Last deploy. At 23:59:53 on 2026-10-22, a run against a4cbc2bd refused: "installed plugin is 4ac0db57, this deploy is a4cbc2bd". The deploy at 00:00:03 on 2026-10-23 then logged DEPLOYED a4cbc2bd -> /home/mohamed/.claude.

Incus fleets

What they are. Three Incus projects on three bridges, owned by OpenTofu (ADR-0077):

• prod default on orch-prod-br, 10.53.1.0/24;
• orchestrata-test on orch-test-br, 10.53.2.0/24;
• orchestrata-dev on orch-dev-br, 10.53.3.0/24.

The project orchestrata (on incusbr0, 10.222.9.0/24) holds infra containers outside that state. See Incus fleets for every instance.

Deploy. From infra/terraform (see infra/README.md):

tofu init -input=false
./import.sh                       # idempotent: skips addresses already in state
tofu plan -detailed-exitcode      # 0 = no change, 2 = drift
tofu apply -input=false
tofu plan -detailed-exitcode      # back to 0

The state is host-owned at /var/lib/orchestrata/terraform/fleet.tfstate, never inside a checkout. The README explains why: "the September state was lost to a worktree gc". The README rule is "A diff means the .tf must change to match reality - never the reverse."

Verification. scripts/check_infra_drift.py runs tofu plan -detailed-exitcode -lock=false. It also proves the three bridge keys that the provider never stores, with incus network get. A missing state is a failure.

The hourly orchestrata-infra-drift.timer writes /var/lib/orchestrata/terraform/drift-latest.json. ship's orchestrata gate also runs scripts/check_env_fleet_agreement.py, which fails when env/*.env drifts from terraform.

Measured. The timer is disabled and inactive. The last verdict, from 2026-10-22T19:01:27+00:00, is drift, plan_exit 2. The cause, recorded on the host services page: boot.autostart was set to false on the test and dev instances, a disk path changed on the test project, and the prod prolog-node-* containers were deleted. The state file was last written on 2026-10-11.

Rollback. There is no rollback mechanism beyond OpenTofu itself. The README's rule is to change the .tf to match reality.

One deploy, verified and rolled back

The sequence below is the codetools path, which ct-install 75 logged end to end on 2026-10-23. It is the cleanest example of the pattern: build, promote, ask the installed command, and roll back if it answers wrong. Hermes follows the same shape with hermes-task verify --live.

Lessons

The config deploy deletes hand-installed hooks. install-claude-config.sh --ref a4cbc2bdf ran at 2026-10-23 00:00. Its rsync -a --delete into ~/.claude/hooks/ removed three hooks that exist in no repo:

• codemap_refresh.sh (SessionStart and SubagentStart);
• precommit_gate_subagent.sh (SubagentStop);
• dispatch_ledger.py (SubagentStop).

settings.json and settings.local.json still registered them. They were restored byte-identical from ~/.claude/.backup-20261023T000002/hooks, and all three are present again. The durable fix is open: commit them into claude/hooks/ and declare their registrations, so the installer deploys them. Until then, check for the three files after every config deploy.

orchestrata-mcp cannot authenticate. Every mcp__orchestrata__* call returns GATEWAY_UNAVAILABLE: GATEWAY_TOKEN is not set. The secrets layer has these keys (names only): DEEPSEEK_API_KEY, GITLAB_TOKEN, OPENROUTER_API_KEY, GEMINI_API_KEY, JWT_SECRET, CONVERSATION_KG_URI and CONVERSATION_KG_DATABASE. There is no GATEWAY_TOKEN. The gateway's default auth mode is JWT, and tokens expire after 3600 s (TOKEN_EXPIRY_SECONDS), so a fixed token in the secrets file would work for one hour. This is an open decision; see open decisions. The prod gateway unit is also disabled, which is a second, independent reason the server has nothing to connect to.

A marker written before its verification lies. On 2026-08-19, ~/.claude/VERSION recorded 19c26470 for five days while Claude ran the c24da189 plugin cache. Since then, install-tools.sh and install-claude-config.sh stage the record and promote it last, and install-orchestrata.sh writes it before restarts but rolls it back on a failed verify.

claude plugin update compares version strings. Same-version content never updates, so the installer reinstalls, and it also compares the cache bytes. On 2026-08-19, ten files in the live plugin cache were stale while the pin looked current.

A plain uv run flips the venv to editable. It caused three refused pushes on 2026-10-22: the pre-push source-provenance gate refused each one. Commit 8a2cb42ad switches the hooks to uv run --no-sync; it is not pushed yet.

The test app container is old. /opt/orchestrata/VERSION inside orchestrata-test:app records 22363abb2, deployed 2026-10-12T13:20:13+00:00. That is 546 commits behind origin/master. Its kernel and gateway units are active.

The provision skill probes a path that no longer exists. Probe 3 of the orchestrata-provision skill checks ~/prod/orchestrata-ultimate. Prod is /opt/orchestrata (ADR-0062).

Related

• ship and deploy
• Deployment sequence
• Deployment inventory
• Host services
• Incus fleets
• codetools deploy
• Open decisions
• Decisions
• Claude hooks
• Gates