Orchestrata Ultimate › Tests

Tests — a guide to the orchestrata test suite

This page explains the orchestrata test suite for a person who has to run it, read its results, or decide whether a red run means anything.

The numbers below were measured on 2026-10-23 in wt-MAP-PRECOMMIT, at 8a2cb42ad on feat/maps-via-codetools, one commit ahead of master (a4cbc2bdf). Where a number comes from another checkout, the page says so.

The generated reference pages hold the per-module detail:

• test reference: every test module;
• run reports: the recorded runs;
• green ledger: which modules are proven green;
• durations: how long each module takes.

1. What the suite is

tests/ holds 2,094 Python files. 1,976 of them are test_*.py modules; the rest are conftest.py files (19 of them), fixtures, helpers and __init__.py. There are 50 top-level directories, and 66 test modules sit directly in tests/.

The generated_* directories

Six directories hold tests that a generator wrote, not a person:

The files are named test_<module>_generated.py. They come from the test generators in src/orchestrata_tools/test_generator/: scripts/gen_tests.py drives EnhancedTestGenerator, and there are Jinja, interface, branch and behavioural generators besides. The generators write interface, branch and behavioural tests against the modules they cover.

tests/generated_recovery and tests/_generated_broken are passed to pytest as --ignore in pyproject.toml, so a normal run never collects them. In September 2026 the generated suites were regenerated with backward-compatible signature pins (see blockers, history section).

Markers

pyproject.toml runs pytest with --strict-markers, so every marker must be declared. Declared markers include:

• unit, integration, no_infra;
• requires_redis, requires_graph, requires_prolog;
• slow, known_slow, llm_integration, incus_live;
• not_parallelizable, isolation;
• a set of REQ_* traceability markers.

The default selection is not slow and not llm_integration and not incus_live. The runner overrides that so slow tests are included, and leaves llm_integration and incus_live as opt-in lanes.

2. How to run it

The runner: scripts/run_tests.py

All suite runs go through one entry point, scripts/run_tests.py. It is not plain pytest. It checks inputs, starts or verifies infrastructure, runs blocking static gates, reuses anything already proven green, then runs the rest in batches across worker processes and writes a machine-readable report.

The run pipeline, read from _main in scripts/run_tests.py through the AST toolkit. Stage numbers are the ones in the source comments.

Modes

From scripts/run_tests.py --help:

Watching a run: pytest-live and observe-pytest

Never start a run and wait for its exit code (RUL-009). A six-minute run gives no signal until the end, and a hang gives none at all. Start the run in the background and watch it:

pytest-live tests/unit -q &        # any project: uses ./.venv/bin/python, puts the observer plugin on PYTHONPATH
observe-pytest list                # instant: running and finished sessions with pass/fail counts
observe-pytest attach latest       # streams PASS/FAIL, exits when the session finishes

Both tools are deployed at /opt/orchestrata-tools/bin. /usr/local/bin/observe-pytest is a symlink to observe_pytest there, and pytest-live is on PATH too. pytest-live also pins OPENBLAS/OMP/MKL_NUM_THREADS=1, because free-threaded Python aborts when native maths thread pools race. See pytest-live observer and the observe-work skill.

In a Claude session the hooks enforce this:

• a foreground test run is denied;
• a full-suite run is denied when the affected-test oracle has bounded the change, and the hook prints the targeted command instead;
• multi-module pytest is rerouted to the threaded observed runner.

3. The rules

For agents: no test run without Mohamed's permission and an agreed time budget. Reading recorded logs and reports is not a run.

4. The environment and the gates

The test fleet

Tests run against the Incus project orchestrata-test on the isolated bridge 10.53.2.0/24. The fleet holds:

• the app container (gateway :8765 and kernel :50053);
• a single Memgraph node (10.53.2.10), plus a disposable HA main/replica pair the HA tests may mutate;
• Redis (10.53.2.11) and Prolog (10.53.2.12);
• the rh-* cluster.

The endpoint profile is env/test.env, with 71 keys and no secrets; secrets layer in from a gitignored file. tests/conftest.py hard-pins the suite to this env and refuses to run if the resolved infra is not on 10.53.2.x. The test env cannot reach prod (Incus ACL). See isolated test fleet.

Since 2026-10-21 the fleet's autostart is off (see the DeepChat review). After a reboot it must be started before integration tests can pass. The runner's infrastructure check (stage 1) reports what is down.

What the pre-push gate runs

scripts/hooks/pre-push, wired through core.hooksPath = scripts/hooks, runs on every push:

1. It refuses to run the suite from a protected-branch worktree.
2. It skips the suite if reports/.ship-verified already certifies this exact source id.
3. It fails closed if the base ref does not resolve.
4. It computes the range from the commit being pushed: BASE...<pushed sha>, three-dot, committed changes only. With an empty diff it exits 0.
5. It runs .venv/bin/python scripts/run_tests.py --affected "$RANGE".
6. On a green run whose report names the pushed HEAD, it writes the source id into reports/.ship-verified, so a re-push of identical source skips the suite.

Affected-test selection on push, from scripts/hooks/pre-push and the runner's stages.

How affected_tests.py decides:

• It walks the serialized import closure, so a changed src/ file selects every transitive dependent test, not only its same-named one.
• It maps changed .pl rule files through a rule-pattern table.
• It always adds the tests/infrastructure smoke floor.
• It names a whole-suite change only when the honest blast radius really is the whole suite: the env/*.env endpoint contract, a src/ file outside the serialized closure, an unrecoverable deleted module, or a module-level change to the root conftest.

Since GATE-AFFECTED (68024751d, 2026-10-21), a push of only inert files (docs, config, generated) skips infrastructure and gates entirely.

RUL-032: green-ledger reuse

The runner reuses proven work at two levels:

• Whole report. If reports/run-tests-latest.json is green, was made at the same head_commit and dirty_sig, and its scope covers this run, the report is reused.
• Per module (the green ledger). Each module has a content key made of the module itself, its src import closure, its conftest chain and pyproject.toml. If that key matches the module's last green run, the module is not re-run, whatever the git identity. Every module that goes green is recorded in reports/test-green-ledger.json. An interrupted run therefore resumes from the modules it already proved.

Today the ledger in wt-MAP-PRECOMMIT holds 9 modules. They come from runs at three commits: 7fa34f61d (2 modules), d4ae44819 (6) and a4cbc2bdf (1). See green ledger.

5. Durations and the slow tail

Most modules are fast, and a few dominate. The components worktree's module duration cache (wt-COMPONENTS/.cache/test_durations.json, last written 2026-10-21) gives:

Why the tail matters. The executor packs modules into bins by their recorded durations (longest first). One slow module caps how fast a bin can finish, and so how fast the whole run can finish. On 2026-10-21 an affected run of 151 modules lost 137 fast modules this way: a slow Memgraph module in the same bin hit the 420 s batch timeout and took the bin down with it. The cache records contention-free durations, so the packer under-provisioned the bins.

How the runner reports slowness. Any test over 60 s in a run is listed as a defect candidate in the summary. A test marked known_slow has had its slowness reviewed, and is listed separately.

The measured floor. The full gate floors at about 250 s on the dev box. See suite parallelism floor and how the suite got 32% faster.

In this worktree the slowest recent single test is test_gen_codebase_map.py::TestDeterminism::test_two_builds_are_byte_identical: 117 s in one of its recorded runs, 19–32 s in the others (reports/test-durations.ndjson, 691 test records over 9 runs). See durations for the generated per-module table.

6. How to read a report

The canonical report is reports/run-tests-latest.json. The runner writes it atomically, so it is always valid JSON. A scoped run or an abort never overwrites a current full-scope record.

Here is today's report, field by field (read with jq):

So this run executed 12 tests in one module, reused 122 tests from 6 modules, and passed. It is evidence for those 7 paths at a4cbc2bdf only.

Checks that tell you whether to trust a green report:

1. test_totals.passed is greater than 0. A run that executed nothing is not green.
2. head_commit is the commit you care about, and dirty_sig matches a clean tree if you need one.
3. scope is "full", or covers the paths you changed.
4. failed and test_totals.failed/errors are 0, and rerun.still_failed is empty.
5. An aborted key means no test ran; its value names the stage that stopped the run.

Before stamping a source as verified, the pre-push hook applies check 1, the head_commit half of check 2, and the failure and error counts of check 4.

Related

• Running the test suite: caveats and teardown patterns
• Test reference (generated)
• Run reports (generated)
• Green ledger (generated)
• Durations (generated)
• Gates, including parallel safety and test purity
• Isolated test fleet
• pytest-live observer
• Rulings
• Workflow
• Research index