--- title: ship and deploy description: What deploy/ship.sh and deploy/deploy.sh each do, step by step published: 1 date: 2026-08-30T18:27:51.000Z tags: deployment, ship, project:orchestrata-ultimate editor: markdown dateCreated: 2026-08-30T18:27:51.000Z --- ## `deploy/deploy.sh` — one call, everything ``` deploy/deploy.sh # deploy at HEAD deploy/deploy.sh --ref # deploy a specific commit deploy/deploy.sh --verify # verify only, change nothing ``` Four things, in dependency order. Each is idempotent and content-compared, so re-running on an unchanged commit converges instead of churning. ### 1. `~/.claude` and the plugin `deploy/install-claude-config.sh --ref ` - `git archive` at the pinned ref into `~/.claude/{hooks,tools,agents,skills,commands,workflow}` — **real files, no symlinks** - its own `uv` venv on free-threaded `python3.14t`, plus `.envrc` for direnv - rewrites every deployed shebang to that venv - installs + enables the **orchestrata plugin** from the **git remote**, pinned to `gitCommitSha` - writes `~/.claude/VERSION` (commit, ref, source, timestamp) — **only after** every check below passes - verifies: files match the ref, shebangs point at the deployed venv, no symlinks, venv is free-threaded, hooks import from it, a hook actually executes, and no deployed file references the source checkout **Refuses** a commit that is not on `origin/master`: the plugin installs from the remote, so an unpushed commit could never be delivered. ### 2. systemd user units `deploy/systemd/install.sh` Installs `gateway`, `kernel`, `ast-live-sync`, `conversation-consumer`, `bot`, and the `kg-retention` / `history-mining` / `orchgraph-backup` timers into `~/.config/systemd/user`. Content-compared, and **restarts what it changed** — `enable --now` does nothing to a unit that is already running, which left daemons on their previous definition. Also restarts when `~/.claude/VERSION` moves, because a unit file can be byte-identical across two commits of the code it runs. ### 3. `/opt/orchestrata-tools` `deploy/install-tools.sh` — the project-agnostic binaries (`ship`, `observe-pytest`, `pytest-live`, `swarm-supervisor`) with their own venv. ### 4. the test fleet `deploy/provision_environments.sh test`, with `ORCH_DEPLOY_REF` pinned. Converges the whole environment — Incus project, bridge, ACLs, containers, orchgraph databases — then deploys the app: - `git archive` at the pinned ref into `/opt/orchestrata` **inside the container** — a copy, never a bind mount - copies the venv (the free-threaded `janus-swi` / `cxpt` / `ctok` extensions are built against the host toolchain, which the read-only `uvpy` + guix mounts still supply) - **excludes `__pycache__`** — cached bytecode records the prefix it was compiled under and produces `ValueError: bad marshal data` at every interpreter start - **repoints the editable install**: `.venv` ships `_editable_impl_orchestrata_ultimate.pth` naming the *checkout's* `src/`, so a copied tree would still import the developer's files - **rewrites shebangs across the whole tree**, not just `.venv/bin` — `orchestrata-kernel-ready.py` is the gateway's `ExecStartPre`, and a stale interpreter path makes systemd report "No such file" about a file that is present - stages as `.incoming` and promotes only when complete - writes `/opt/orchestrata/VERSION` Then refuses to finish if any reference to the source checkout survives in the deployed venv, and asserts `orchestrata_ultimate` imports from `/opt/orchestrata`. Skipped when no Incus daemon is reachable — no fleet is not a failed deploy. A fleet that is **present** and refuses is fatal. ### 5. verify Re-runs itself with `--verify`, which **asks each deployed thing what it is**: `~/.claude`'s verification, and the container's `/opt/orchestrata/VERSION` compared against the expected SHA. Probe, never trust (ADR-0053). ### Not here **Prod.** `scripts/promote_env.py` owns promotion; a deploy of a feature branch must never rewrite the production fleet as a side effect. **dev** is developer scratch, converged on demand. --- ## `deploy/ship.sh` — verified is the only gate ``` bash deploy/ship.sh # gate, push, deploy, verify, promote bash deploy/ship.sh --no-verify # skip the gate (already run this session) bash deploy/ship.sh --dry-run # print the plan, change nothing bash deploy/ship.sh --no-promote # stop before prod promotion ``` ### Before anything - **self-bootstrap**: if the repo carries its own `deploy/ship.sh` and the running process is a different file (the `/opt` copy), it hands off to the checkout's version — otherwise a broken gate could never fix itself. Terminates after one hop. - **refuses a detached HEAD** - **refuses a dirty tree** — uncommitted source is "still working", and shipping it would push a commit that differs from what was verified. Generated artifacts are excluded (`docs/CODEBASE_MAP*.md`, `plans/ast-*.json`, `reports/*`, `.claude/run/*`, `dr-hannah/data*`), because the tooling rewrites them as it works. ### 1. verify Each project states its own gate; nothing is inferred from a file listing. For **orchestrata-ultimate-architecture**: 1. `ruff check src/ tests/ scripts/ claude/` 2. `black --check src/ tests/ scripts/ claude/` — black, not `ruff format`; this repo formats with black by decision 3. `check_ai_attribution.py --range @{u}..HEAD` (RUL-007) 4. `run_tests.py --affected ` — the authoritative suite, scoped to the blast radius **Run-evidence reuse.** A stamp at `reports/.ship-verified` holds ` `, where the source id is a SHA-256 over the working-tree content of every tracked file outside `reports/`. Re-running the suite for source that already passed is waste; reusing a result for source that *changed* would report a green verification for a suite that never saw the code. The stamp only counts when the recorded run was actually green — a stamp alone is not evidence. ### 2. push `git push origin HEAD`. The **pre-push hook** verifies the blast radius of the range git says is being pushed (`...`), not `origin/master...HEAD` — those differ whenever the pushed ref is not the checked-out branch. Never force-pushes, never rewrites history, never deletes a remote branch. ### 2b. merge to master For this repo on a non-master branch: fetches `origin/master`, **refuses if the branch is behind** (a merge here would guess at a reconciliation), then fast-forwards master. ### 3. deploy `bash deploy/deploy.sh --ref $(git rev-parse HEAD)` — one call, everything above. ### 4. verify deployment `bash deploy/deploy.sh --ref --verify`. Fails if the fleet is not running the commit this ship pushed. ### 5. promote prod `promote_env.py promote --to prod --yes --skip-ingest`. `--yes` because ship *is* the deliberate path; the interlock exists to stop a bare hand-typed promote. ### 6. wire For knowledge-wiki: `wiki-rag-ingest`, because content that is not indexed is invisible to every RAG query — the same defect as a commit that is never pushed. --- ## Why it is shaped this way Every rule above is a scar. - **`ship` exists** because "push and deploy when verified" as a *rule* depends on someone remembering, which is exactly the failure mode. It is a mechanism instead. - **`deploy.sh` exists** because the deployment was four scripts in a remembered order, so "deployed" meant whichever had been run. Measured 2026-08-30: ship called `install-claude-config.sh` alone, so every ship reported a completed deploy while the kernel ran the developer's bind-mounted checkout, the user units waited for a manual install, and `/opt/orchestrata-tools` drifted. - **Code is copied, never linked.** The app container used to bind-mount the checkout: the kernel's `ExecStart` was the checkout's interpreter, and the adapter file inside the container shared an inode with the host's — the same file, from a branch with 109 uncommitted edits. Nothing was pinned, versioned or auditable, and every "the fleet is green" claim described that dirty tree. - **Verification asks, it does not trust.** A verifier that reads the deployer's own output proves only that the deployer ran.