Orchestrata Ultimate › Gates › Suite preflight

Gate family: Suite preflight

At a glance

Every authoritative test run goes through python scripts/run_tests.py, and before a single test executes, that runner does two kinds of gating:

1. Infrastructure preflight (scripts/start_test_services.py) — is the real fleet actually up and wired?
2. Static safety gates (run_tests.py::_run_gates) — five blocking, read-only analyses that refuse to start a run that would produce a worthless or flaky verdict.

Both exist for the same reason: a suite verdict is only evidence about the world it actually ran against. A run against the wrong checkout, an unwired kernel, or a parallel-unsafe test corpus produces a green or red that means nothing.

1. Infrastructure preflight — scripts/start_test_services.py

From its module docstring:

"Checks all four services (Redis, Graph, Gateway, Kernel) in parallel. Starts any that are down, verifies each with a functional test, and prints a colour-coded report. Idempotent — safe to run any number of times.

A fifth, non-parallel check (Link) runs after the four above have settled: it asks the gateway, over its real WebSocket transport, whether it is actually wired to the kernel over gRPC (gateway.info's "kernel" block) — because 'the gateway answers gateway.ping' and 'the kernel accepts a connection' can both be true while the gateway is silently running against an in-process fallback with zero real traffic reaching the kernel."

python scripts/start_test_services.py            # Check + start + verify
python scripts/start_test_services.py --status   # Check only (no start)
python scripts/start_test_services.py --json     # JSON report
python scripts/start_test_services.py --skip KERNEL,GATEWAY

The kernel is REQUIRED. Project law (CLAUDE.md Key Rules, docs/kernel-is-required.md): the Agent Runtime kernel (runtime_main, gRPC on CXP_GRPC_PORT/50053) is marked required=True by this script; its failure aborts the run. The only sanctioned opt-out is an explicit --skip kernel, which makes the run non-authoritative — a suite run that does not start and gRPC-verify the kernel is invalid/false-green, because kernel-dependent paths go silently untested.

2. The five static safety gates — run_tests.py::_run_gates

From the function's own docstring: "Run blocking static gates… all BLOCKING (user ruling 2026-07-10: no violations allowed)… All … gates are independent, read-only static analyses with disjoint outputs, so they run CONCURRENTLY (a ThreadPoolExecutor around the subprocess.run calls…). Still fully blocking: any gate failing fails the run."

An honesty note on counting: the docstring enumerates four gates, but the code's gates list carries five — source provenance was added at the head of the list after the docstring was written, with its own inline rationale ("First, because every other gate's verdict is meaningless if the suite is about to import a different checkout's source than the one it is testing").

--skip-gates skips this stage entirely (for iteration, never for an authoritative run).

2.1 source-provenance — scripts/check_source_provenance.py

Refuses: a suite run whose imported package was not installed from this checkout, or resolves anywhere but site-packages.

Updated 2026-10-23. Since 9b4ac72b4 (2026-10-21, "proper-install: re-base the source-provenance gate on PEP 610 direct_url.json") the "resolves to src/" check described in the rest of this section is gone; _install_origin()'s docstring calls it "permanently false" now that the repo installs non-editable into site-packages. main() makes two checks (read through the AST toolkit):

1. Install origin. The url in the orchestrata-ultimate distribution's PEP 610 direct_url.json must equal this checkout's file:// URI. Otherwise it prints "THE SUITE WOULD TEST THE WRONG SOURCE TREE", the installed from origin, and "Another checkout re-synced this venv. Fix: run uv sync --dev --no-editable from THIS checkout." and exits 1.
2. Resolves under site-packages. The imported package's directory must have site-packages among its path parts. Otherwise it prints expected source and actual source, any shadowing sys.path entries and PYTHONPATH, and exits 1. When the two source lines are identical, this checkout's own src/ won the import — the editable-install case a plain uv run re-sync produces (derived from the code).

The full gate page is source-provenance.

Why it exists (docstring, verbatim incident): "On 2026-07-27 the 'final gate' run in a worktree reported 30 failures that were pure artifacts: an exported PYTHONPATH=/home/mohamed/workspace/orchestrata-ultimate-architecture/src leaked into the run and landed at sys.path[1], outranking the worktree's own editable .pth entry at sys.path[6]. The run executed the worktree's TESTS against a DIFFERENT BRANCH's SOURCE. Both directions of that verdict were worthless."

How it runs: imports the package the way the suite does, resolves where it came from, fails when it did not come from this repository's src/. Exit 0 = resolves inside this checkout; exit 1 = resolves elsewhere or cannot be imported at all. It is deliberately process-global (it verifies THIS interpreter's import path), so it takes no target-tree argument.

Tests: tests/scripts/test_check_source_provenance.py, tests/scripts/test_check_source_provenance_errors.py.

2.2 parallel-safety — scripts/gate_parallel_safety.py (sub-gates 0.1–0.6)

Also a registered battery gate — full detail on its own page. Summary of the sub-gates, from the script's own section markers:

Waivers: a finding on a line carrying (or directly below) # gate-ok: <reason> is suppressed; "a reason is required; a bare # gate-ok: does not waive." In the preflight, the runner can point it at a target tree via --tests-dir (integration- test seam only; production always analyzes the full tree).

Tests: tests/scripts/test_gate_parallel_safety.py.

2.3 test-purity — scripts/audit_test_purity.py

Also a registered battery gate — full detail on its own page. In the preflight it runs over tests/ and blocks on any mock/fake/stub/monkeypatch/skip pattern. The _run_gates docstring records why it joined the preflight: it "previously only ran inside the manual health_check, so its findings could not block a suite run."

Tests: tests/scripts/test_audit_test_purity.py.

2.4 prolog-boundary — scripts/check_prolog_boundary.py

Also a registered battery gate — full detail on its own page. Preflight role (per _run_gates): "every Python Prolog query site must target a predicate/arity that exists in the consulted rule base" (reports/proposal-wire-prolog-governance.md §7 Phase 1).

Tests: tests/scripts/test_check_prolog_boundary.py.

2.5 rulings-lint — scripts/rulings_lint.py

Refuses: a ruling in the knowledge-wiki rulings canon that declares no real enforcement mechanism.

Why it exists (docstring): "Rulings enforcement lint — closes the prose-only loophole (SPEC-023 Phase 2)… A ruling with no ## Enforcement section, or with a declared mechanism that does not actually exist, is a FAILURE (exit 1) — this is the mechanism that keeps a ruling from silently regressing to prose-only (RUL-014)."

How it runs: reads rulings/index.md + rulings/RUL-*.md from the git-backed wiki store on the filesystem (default ~/workspace/knowledge-wiki/rulings, override via RULINGS_DIR) — "a live Wiki.js server must never be a dependency of this gate." Each ruling's ## Enforcement bullets must match one of five grammars:

- hook <path>                    file must exist and be executable
- gate <script> <rule-id>        script must exist AND <rule-id> must be a literal substring of the script text
- banned-deps <name>             <name> must be a (non-comment) line in .claude/banned-libraries.txt
- constitution <article-text>    literal substring of src/orchestrata_ultimate/prolog/constitutional.pl
- none (judgment)                always passes; reported as judgment-only

Failure semantics: exit 1 on a prose-only or falsely-declared ruling. Missing RULINGS_DIR is a SKIP, not a FAIL (exit 0 with a warning): "a machine that has not cloned knowledge-wiki must not go false-red on this gate — same honesty rule as --skip kernel."

Wiring beyond the preflight: scripts/check_wiring.py (W4) imports this module's rulings-directory resolution to find which scripts a ruling declares as its gate mechanism; scripts/health_check.py also invokes it.

Tests: tests/scripts/test_rulings_lint.py.

Typical failures and how to fix them

• Any of the five static gates failing fails the run before any test executes — the runner prints each gate's stdout and stops. Fix the finding the gate names; each gate also runs alone for a quick loop.
• A gate script missing from disk prints ⚠ <script>.py not found — skipping gate rather than crashing (each gate is checked for existence before dispatch). Treat that line as a defect, not a pass.
• Infra preflight failure on a required service (the kernel above all) aborts the run; --skip kernel produces a run you must label non-authoritative.
• source-provenance: "Another checkout re-synced this venv". Run uv sync --dev --no-editable from this checkout, then re-run.
• source-provenance: expected and actual source identical. The venv was re-synced editable (a plain uv run does this); re-sync with --no-editable. With a PYTHONPATH printed, unset it: "the proper install needs no PYTHONPATH."
• --affected: REFUSED — cannot bound the blast radius. At least one changed file has no resolution. Supply --affected-decision '<citation>' to proceed with the bounded remainder, or fix the missing fact the message names (RUL-040).
• A run killed at the cap. The test phase stops at ORCH_TEST_CAP_SECONDS (default 300). Modules that went green before the deadline are in the green ledger, so a re-run continues from there.
• "N still failed" after the solo retry. Those failures reproduced alone and are real. Recovered modules are load/order flakes, reported and appended to .flake_recoveries.ndjson; over the cap (default 25) modules are left UNVERIFIED.

4. The --affected (pre-push) path — classification now runs before infra

Updated 2026-10-23. The order described in the table below was the state on 2026-10-21. Commit 68024751d (2026-10-21, "GATE-AFFECTED — classify affected files before infra so pure non-code pushes skip the service stack", on origin/master) moved affected resolution to step 0.75, ahead of the infra preflight. _main now runs, in order: 0 freshness guard; 0.5 basetemp prune; 0.75 _resolve_affected() → _plan_affected_run(); 1 infrastructure; 2 safety gates; 2c the RUL-032 green ledger; 3 the test phase; 4 report. At 0.75: an unbounded change with no --affected-decision aborts as REFUSED; if _affected_all_inert(resolved) is true the run prints "--affected: all changes inert (non-code) — no services or tests required; passing.", writes a canonical report with "non_code": true and the note "all changed files inert (non-code) — skipped infra + gates + smoke floor", and returns 0. The test phase prints "Selected: N tests in M modules (K modules reused from ledger) · hard cap 300s" with the default cap. The "known inefficiency" paragraph below is therefore closed. Read from run_tests.py::_main through the AST toolkit on 2026-10-23.

The pre-push hook invokes run_tests.py --affected <range> (the RUL-040 blast-radius gate). That path runs the SAME preflight as a full run, then resolves the changed files to a test subset. The order, measured in run_tests.py::_main (2026-10-21, before 68024751d):

The classification (2b) is pure git + static analysis and needs no services — yet it runs after the infra preflight (1). So a pure non-code push (docs, skills, reports, config) still boots the kernel + graph + gateway and provisions all worker databases, then resolves the files as INERT and runs only a smoke floor.

Known inefficiency (filed 2026-10-21, closed by 68024751d the same day): a pure non-code push could short-circuit — run affected_tests.py first; if every changed file is INERT, skip infra (step 1), the static gates (step 2), and the smoke floor entirely, and pass. The reordering is a small change to run_tests.py::_main (move step 2b ahead of step 1). Tracked in plans/briefs/gate-affected-classify-before-infra.md.

Related

• Gates master index
• Development workflow
• source-provenance, rulings-lint, freshness-guard, affected-selection, green-ledger, test-phase-cap, rehearsal-exclusion, pre-push-guards
• RUL-032 — Tests run once per code state
• RUL-040 — Scope measured at symbol granularity, never full suite
• RUL-038 — Test runs never create containers and RUL-039 — the rehearsal class is never in the promotion gate
• ADR-0111, ADR-0120
• Registered-gate pages for the three dual-role gates: parallel-safety, test-purity, prolog-boundary
• .claude session hooks — no-xdist.sh and force-test-observation.sh gate how the suite may be invoked; this family gates what happens once it is.
• Runner tests: tests/scripts/test_run_tests.py, tests/scripts/test_run_tests_main_infra.py, tests/scripts/test_start_test_services.py, tests/scripts/test_start_test_services_coverage.py.