Orchestrata Ultimate › Gates › ship-verify

Gate: ship-verify

At a glance


Purpose	"Verified is the only gate; once it passes, push and deploy and re-verify happen without a decision in between" (`deploy/ship.sh` header). It "refuses rather than guesses: an unknown project, a dirty tree, a detached HEAD or a failing gate stops it with a reason."
Where it runs	`deploy/ship.sh` (810 lines), step 1 `verify`, then step 2a (evidence commit) and step 2 (push). `ship` on PATH is `/usr/local/bin/ship`, byte-identical to the repo copy when read (2026-10-23); the worktree copy has no diff against `origin/master`. Last change `7b859bb4a` (2026-10-22).
Trigger	A human or agent runs `ship` in a repo; the unshipped-guard names it as the remedy at every turn end with local-only commits.
Exact command	`ship` · `bash deploy/ship.sh` · `--dry-run` (print the plan; `run` only echoes) · `--no-verify` (skip this script's gate, not git's hook) · `--no-promote` (skip step 5).
Blocks	`FAIL: <reason>` on stderr, exit 1, at the first failing step. Unknown argument: exit 2.
Ruling / ADR	RUL-007 (attribution range) · RUL-032 (one run per code state: the source-id reuse) · RUL-038 (`gate_pytest` is a sanctioned runner) · ADR-0077 item 4 (env files match terraform) · ADR-0062 (prod root) · ADR-0087 (the prod gate).

Before the gate: refusals

Read from deploy/ship.sh lines 29–140:

Not a git repo → FAIL: not inside a git repository.
Self-bootstrap. When the repo carries its own deploy/ship.sh and it differs from the running file, ship execs the repo copy ("the checkout's copy governs how this repo ships"). One hop. Cause: on 2026-08-14 the deployed copy stopped at its own broken formatter gate before reaching the deploy step that would have installed the fix.
Project name comes from the origin remote, not the directory (2026-09-22: the checkout ~/workspace/orchestrata-ultimate was reported as "no gate defined" although its remote, orchestrata-ultimate-architecture, has one).
Detached HEAD → FAIL: detached HEAD — check out a branch first.
Dirty source → FAIL: working tree has uncommitted SOURCE changes. Excluded as generated: docs/CODEBASE_MAP.md, docs/CODEBASE_MAP_FULL.md, plans/ast-*.json, reports/*, .claude/run/*, and the wiki's two auto-refreshed live-data paths.

The verify arms

Project (`NAME`)	Gate
`orchestrata-ultimate-architecture`	The sequence below
`knowledge-wiki`	`gate_pytest timeout 300 .venv/bin/python -m pytest tests/ -q`, `ruff check rag/ tests/`, `ruff format --check rag/ tests/`
`dr-hannah`	`gate_pytest timeout 900 uv run python -m pytest tests/ -q` (369.75 s measured 2026-08-14, so 300 s failed on wall clock), `uv run ruff check`, `uv run ruff format --check` on `src/ tests/`
anything else	`FAIL: no gate defined for '<name>' — add one to deploy/ship.sh rather than shipping unverified`

gate_pytest runs its command under env ORCHESTRATA_SANCTIONED_RUNNER=ship.sh. A Claude session loads the RUL-038 pytest observer into every pytest, and the observer stops a session spanning more than one test file outside a sanctioned runner. Measured 2026-10-22: the wiki's ship failed at collection from a Claude session ("selected tests from 13 files outside a sanctioned runner") and passed from a terminal; fixed in 7b859bb4a.

The orchestrata arm

uv run ruff check src/ tests/ scripts/ claude/.
uv run black --check src/ tests/ scripts/ claude/ — black, not ruff format: the two disagree, and asserting ruff's style "demanded 439 files be reformatted away from the project's own style".
RUL-007: uv run python scripts/check_ai_attribution.py --range "<base>..HEAD", base @{u}, or origin/master after a fetch when the branch has no upstream (2026-10-06: "fatal: no upstream configured" had been reported as an attribution failure).
The suite — see the next section.
uv run python claude/scripts/gen_plugin.py --check — "<1s". Added after a claude/hooks fix shipped on 2026-08-16 while the generated plugin mirror, the copy the plugin cache executes, stayed stale.
uv run python scripts/check_env_fleet_agreement.py — env/test.env, env/dev.env and .env.example's TEST_* keys must match terraform (ADR-0077 item 4). Added after the 2026-10-08 recorded run found "a two-month-stale prod-subnet TEST_MEMGRAPH_URI that nothing checked".
PUSH_NO_VERIFY=1.

The comment above step 1 (lines 223–226) still says the pre-push hook runs the authoritative suite; the code below it (lines 251–272) runs the suite here and pushes with --no-verify. The second comment matches the code.

The suite, and when it is skipped

Why it runs inside ship, not in the hook. git push opens the SSH session before the hook runs. On 2026-08-14 the hook escalated to all 1630 modules (489 s) and GitLab closed the idle session twice, after a fully green suite (33091 tests passed, 0 failed · 1630/1630 in 489s / Connection to gitlab.com closed by remote host.); ServerAliveInterval did not help. So ship runs the hook's work itself and pushes with --no-verify. A plain git push still runs the hook.

Reuse (RUL-032). ship_source_id hashes the working-tree content of every tracked file outside reports/ plus the names of untracked files. The stamp reports/.ship-verified holds <source-id> <commit>. ship_already_verified skips the suite only when all hold:

stamp field 1 equals the current source id;
reports/run-tests-latest.json exists with failed == 0, test_totals.failed == 0, test_totals.errors == 0, test_totals.passed > 0;
the report's head_commit equals the stamp's commit (added after 2026-08-19, when a 10-module hand-run subset stood in for the whole suite).

It prints authoritative suite SKIPPED — this exact source already passed.

Blast radius base. Otherwise the suite runs from the last verified commit, never from origin/master (2026-08-19: origin/master...HEAD was empty after a re-ship and selected only tests/infrastructure, which would have stamped five unverified commits). Base = stamp field 2, else the newest commit titled record the authoritative run for <sha>; with neither, the whole suite runs: "Slow and correct beats fast and vacuous." The stamp is written only after a pass and never under --dry-run (2026-10-08: a dry run had stamped its own HEAD, which would have certified 21 unverified commits).

State read 2026-10-23 (wt-MAP-PRECOMMIT tree, all three files tracked): the stamp's commit is a4cbc2bdf; run-tests-latest.json has head_commit a4cbc2bdf, failed: 0, 134 passed, and affected: "subset run — non-authoritative, full suite still owed". ship_already_verified does not read the affected field, so the runner's own "non-authoritative" label does not stop a skip. Whether a hand-run subset at the stamped commit can therefore satisfy the skip was not tested.

Where the full suite runs. Step 5 (promote prod) runs uv run python scripts/promote_env.py test test, which "runs scripts/run_tests.py --json over the whole suite (step 1 ran only --affected, so it fills nothing this step can reuse)" and records the PASS the prod gate requires (comment, lines 698–709; ADR-0087, ADR-0088). An orchestrata ship with promotion therefore runs an affected suite in step 1 and a whole-suite run in step 5. Their combined wall time is not measured; the comments record one full ship at about 18 minutes on 2026-08-14, 15+ of them re-ingest, since dropped with --skip-ingest.

Step 2a: commit the run evidence

For orchestrata, outside --dry-run: the suite rewrites reports/run-tests-latest.json, so ship commits it and the stamp as record the authoritative run for <sha> (for example 4ac0db570, 796c4f0b5, e4a1be70d, all 2026-10-22). Any other change outside the generated list fails with tree changed during the gate (beyond run evidence). Origins: on 2026-08-14 a green 33k-test ship died at the merge with "Your local changes would be overwritten by checkout"; on 2026-08-19 a green run died on regenerated CODEBASE_MAP files before the generated list was applied here.

Step 2: the push decision

AHEAD is recounted after 2a (2026-09-23, unit orch-ship14: the banner said 0, 2a created a commit, the push was skipped, and step 3 refused "not on origin/master"). 0 → no push. Gate ran here → git push --no-verify origin HEAD. Otherwise → git push origin HEAD, which runs the pre-push guards.

--no-verify skips only this script's gate. The script says so and warns when a pre-push hook is installed: on 2026-08-14 a ship --no-verify "announced 'gate skipped' then sat in a ten-minute suite".

After the gate

Not gates of this page, listed for orientation (deploy/ship.sh lines 504–809; detail on ship and deploy): merge to master (fast-forward only, via deploy/ship-merge-master.sh; refuses when the branch is behind), deploy (each known installer; an unknown installer in deploy/ is a refusal), wire (re-index RAG) for the wiki, verify deployment (install-claude-config.sh --verify), and promote prod (5a: /opt/orchestrata HEAD equals origin/master; 5b: every orchestrata-/orchgraph- unit executing from it is active).

Typical failures and how to fix them

Message	Cause	Fix
`working tree has uncommitted SOURCE changes`	Uncommitted source	Commit it; it is part of the change
`ruff check` / `black --check`	Lint or format drift	`uv run black <files>`; fix the ruff finding
`AI-attribution gate (RUL-007)`	A commit in the push range carries AI attribution	Reword the commit; see AI attribution
`run_tests (the authoritative suite)`	A red test, or a runner abort (freshness, rehearsal artifact, infra)	Read the run's first failure; see freshness guard, rehearsal exclusion
`generated plugin drifted`	`claude/` changed, `plugins/orchestrata` not regenerated	`uv run python claude/scripts/gen_plugin.py`, commit
`env files drifted from terraform`	Hand edit or stale env file	`uv run python infra/gen_env.py --profile test --write env/test.env`
`tree changed during the gate`	Something wrote source while the suite ran	Find the writer; re-ship on a quiet tree
`pytest` (wiki arm) from a Claude session, 0 tests run	RUL-038 observer, before `7b859bb4a`	Use the current ship
`no gate defined for '<name>'`	A repo ship has no arm for	Add an arm to `deploy/ship.sh`

Gates index
Workflow
Unshipped guard
Pre-push guards
Affected-test selection
Green-ledger reuse
Plugin drift
Env subnet scheme (no gate page covers check_env_fleet_agreement.py yet)
CI pipeline
Ship and deploy · Deployment