Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0087

ADR-0087: The prod gate is one correctness run of the promoted commit

field	value
Status	Accepted
Date	2026-10-11
Register	factory (`docs/adr`, 4-digit ids)
Source	`docs/adr/0087-the-prod-gate-is-one-correctness-run-of-the-promoted-commit.md` — permalink at `07aa3b3c6` (unchanged on master since that commit)
Wiki register entry	ADR-0087 (wiki status: Accepted)

← ADR-0086 · ADR-0088 →

The record

Status: Accepted · Date: 2026-10-11 · Supersedes: ADR-0079

Context

The 2026-10-11 promote of master e4266420f (conversation-recall fix) spent about two hours and never reached prod. Measured from scratchpad/ship-graph-defects-2.log and scratchpad/promote-prod.log:

Finding	Measurement
R1 ship's promote arm runs `promote_env.py test test` believing "step 1 proved the suite green … seconds, no re-execution"	step 1 (verify) ran `--affected`: 2 modules, printed "non-authoritative, full suite still owed"; `test test` then ran 1,680 modules (~55 min). After ship's run was killed, the same suite was started a second time by hand.
R2 benchmarks gate the promote	~23 modules (`_perf`, `_performance`, `test_benchmark_*`, `session_stress`, `strict_targets`, `system_limits`, `flaky_test_hunter`) measure the box, not the code; `test_workspace_perf::test_parallel_parse` failed under load and passed at load 14 (55 s) with no code change
R3 every `run_tests.py` start runs a deep pollution sweep of the primary `orchkg`	725,636 nodes; basic count 71 s + deep count 278 s; result "clean"; the bare scan floor is 9.3 s. Five suite tests launch a nested real `run_tests.py`, each repeating the sweep inside a 120 s test timeout — they fail on that alone (tiny run: 6 min 15 s startup, 3 s of tests)
R4 the rerun round repeats budget casualties under the same load	523 modules had no result after the first pass; 425 never ran (batch `session-timeout` 273 s exhausted). The rerun round re-ran them 8-wide; 429 still failed
R5 (ADR-0079, Proposed, never built) the gate accepts any `PASS` record	`gate_for("prod")` checks `result == "PASS"` only; the record carries no commit

Decision (Mohamed, 2026-10-11: "yes")

One correctness run, bound to the commit. promote_env.py test <env> records the commit (git rev-parse HEAD) and ship's source id with the result. gate_for("prod") refuses unless the latest test-env record is PASS and its sha equals the commit being promoted; a record without sha is treated as absent. ship's test test step is that run, and its comment states so (no "seconds" claim).
Benchmarks leave the gate, not the suite. Performance/stress modules are declared once (a perf lane list read by the runner). Gating runs exclude them; run_tests.py --lane perf runs them and records the result, non-gating. Nothing is skipped: every perf test still runs, in its lane.
The deep sweep runs once per suite, fast. The outermost run_tests.py sweeps and exports a marker; a nested run_tests.py / start_test_services.py that sees the marker does not sweep again. The sweep's Cypher keeps its exact match set and puts every regex behind the cheap CONTAINS/STARTS WITH predicate it implies (deep count 278 s → 112 s measured; further gains only with an equivalence test).
Reruns re-run failures, alone. After the first pass, modules that never ran (budget casualties) are scheduled into fresh batches and are not counted as failures; modules that ran and failed are re-run one at a time.

Alternatives rejected

Keep the full suite including benchmarks as the gate — rejected: a load-dependent wall-clock threshold blocks prod for reasons unrelated to the commit (R2).
Raise test timeouts — rejected: hides R3 and R4 instead of removing them.
Gate on the affected subset only — rejected: the runner itself labels it non-authoritative.
A freshness window on the PASS record — rejected in ADR-0079: time is not identity.

Consequences

A prod promote needs exactly one full correctness run, of the promoted commit; a stale PASS cannot admit a different tree.
Perf regressions are still measured, in their own lane, without blocking prod.
Suite startup drops by the sweep cost for every nested runner test.
reports/run-tests-latest.json distinguishes "failed" from "never ran".

Provenance

Mohamed, 2026-10-11: "investigate if we are doing anything redundant here for example promotion does not require performance benchmarking and there were test running and reruning unnecessarily", then "fix the sweep, then promote", then "yes" to the four-part recommendation above.
Supersedes ADR-0079 (Proposed 2026-10-08), whose commit-binding decision is item 1 here; its "ship gains no test test step" point no longer matches ship, which already runs it.
Measurements: this session's logs listed in Context; sweep timings from read-only counts against the test fleet orchkg (10.53.2.10).

Supersedes: ADR-0079

Rulings cited

None.

Specs cited

None (or no spec page exists for the ones named).