Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0079

ADR-0079: The prod interlock requires a green test-env record of the commit being promoted

← ADR-0078 · ADR-0080 →

The record

Status: Superseded by ADR-0087 (2026-10-11) · Date: 2026-10-08

Context

scripts/promote_env.py promote --to prod is gated by Promoter.gate_for("prod") (src/orchestrata_ultimate/promotion/promoter.py), which reads the promotion ledger and refuses unless the test env's latest test record has result == "PASS". That record is written by promote_env.py test test and carries a timestamp and a return code — not the commit it tested. The gate therefore accepts ANY latest green record, whatever tree produced it.

Measured 2026-10-08, shipping feat/recorded-run-20261008-redesign (25 commits over origin/master 1d5f3e52c):

• The ledger held test=PASS@2026-10-08T10:27:27Z, recorded for 1d5f3e52c before that morning's prod promote (promote=DEPLOYED@10:46:51Z sha=1d5f3e52c6a4).
• deploy/ship.sh's promote arm (promote_env.py promote --to prod --yes --skip-ingest) would have been admitted by that record while promoting a32b14194 — a tree the recorded run never saw. The only thing standing between that and a prod deploy of unverified code was ship's own affected suite (a subset by design) and its post-hoc 5a identity check, which proves the tree ADVANCED, not that it was TESTED in the test env.
• The honest path taken instead: promote_env.py test test re-run on the new tree before the ship, as a user systemd unit observed live.

The promote record already carries sha; the test record is the one that does not. ship.sh's own run-evidence stamp (reports/.ship-verified, ADR-era fix 1f02eb894) solved the identical problem one layer up by binding the evidence to a content hash AND the commit — the ledger never caught up.

Decision (proposed)

1. promote_env.py test <env> records, with the result, the commit it ran at (git rev-parse HEAD of the checkout it ran from) and ship's source id (ship_source_id, the content hash of every tracked file outside reports/), so the record names exactly the tree the suite saw.
2. Promoter.gate_for("prod") refuses unless the latest test-env record is PASS and its sha equals the commit being promoted (the resolved target of the promote, which ship.sh makes origin/master after the fast-forward). A record with no sha (pre-ruling ledger) is treated as absent — the migration path is to run test test once.
3. deploy/ship.sh's promote arm keeps its post-hoc 5a/5b checks; this ruling closes the pre-hoc gap only. ship.sh does not gain a test test step of its own: the recorded full run stays the explicit, observed command the RUL-032 procedure already names, and the interlock now proves it happened for this tree.
4. run_tests.py's aborted: freshness outcome (observed the same day when swipl was absent from a minimal systemd-run PATH) prints the freshness guard's cause on stdout and in reports/run-tests-latest.json, so an abort is never a bare word.

Alternatives rejected

• Keep the gate as it is — rejected: the case above is a real false-green path; "the last green run" is not evidence for a different tree.
• A freshness window on the record (e.g. ≤ 1 h old) — rejected: time is not identity; a commit can land one minute after a green run.
• Compare the source id only, not the commit — rejected: the source id proves content but the promotion ledger, the prod tree and ship's 5a check all speak in commits; the record carries both so either side can be checked.
• Make ship.sh run test test before promoting — rejected for now: it would double the suite inside one ship (verify already runs the affected set) and hide the recorded full run behind another script; the interlock is the right place to refuse.

Consequences

• A prod promote is impossible without a green test-env run of that tree; ship.sh fails at "promote prod" with the record's sha and the target sha named in the refusal.
• The ledger grows a sha/source_id on every test record; status shows them so drift between "tested" and "deployed" is visible at a glance.
• Enforced by: tests/scripts/test_promote_env.py::test_gate_prod_requires_green_test_run extended to the sha comparison; a new test that a green record for another commit is refused; tests/deploy/test_ship_master.py promote-arm tests unchanged.
• Until ruled and built, the operator rule stands: run promote_env.py test test on the exact tree before any ship that promotes.

Provenance

• Found by Claude Code on 2026-10-08 while finishing the eight-failures swarm (plans/recorded-run-2026-10-08-eight-failures-design.md §8.1): the ledger (promote_env.py status) read test=PASS@10:27:27Z for 1d5f3e52c while the tree to promote was a32b14194; Promoter.gate_for source confirmed the record is compared on result alone.
• Mohamed's standing rule for ship (deploy/ship.sh header): "Verified is the only gate; once it passes, push and deploy and re-verify happen without a decision in between." A gate that accepts another tree's verification does not meet it.
• Related: ADR-0062 (prod tree is /opt/orchestrata), ADR-0067 (tool deployment identity), ship.sh run-evidence stamp fix 1f02eb894.

Related ADRs

• Superseded by: ADR-0087
• Also mentioned: ADR-0062, ADR-0067

Rulings cited

• RUL-032: RUL-032 — A test runs once per code state; escalation never re-runs a proven test

Specs cited

None (or no spec page exists for the ones named).