Orchestrata Ultimate › How we work

How we work — process, deployment, testing, coverage

This is the story of how a change actually travels through orchestrata-ultimate: who designs it, who writes it, what it has to survive before it is allowed to land, and where it runs once it does. It ties together the rules scattered across AGENTS.md, CLAUDE.md, docs/LEVEL0.md, and the kernel/Prolog design notes into one narrative. If you only read one page to understand the working discipline of this project, read this one, then follow the links for depth:

• Workflow, the step-by-step path with diagrams and a worked example;
• Gates — master index, every gate that can say no;
• Running the test suite and Reproducibility & IaC.

How we work — who briefs, who writes code, one authoritative source

The project runs on a division of labour that Mohamed made explicit on 2026-10-22 in RUL-041 (ADR-0125):

"this claude here remains production claude but it writes no code when working on my project just perfects the briefs checks and workflow with the flagship model while measuring sonnect and qwen and deepseek performance and workflow improvements"

• The host Claude (the flagship model, on the host) writes no orchestrata code. "Code is anything executed, imported or loaded": source, tests, scripts, hooks, systemd units, pyproject.toml and uv.lock, terraform, CI. Its work is the briefs, checking each worker's result by execution, and improving the template, the improvement log and the mechanisms that catch repeated defects.
• The worker models write the code: Sonnet (Claude subagents, the default for delegated work under RUL-026), Qwen and DeepSeek (Hermes, pi-task).
• Every dispatch is measured: model, time, cost, the first-pass result of the checks, the defects found and the rework rounds. A workflow change counts as an improvement only if those numbers improve. The SubagentStop hook records each finished worker in the Dispatch ledger; the changes and their numbers are on Workflow improvements.

This replaces the earlier description on this page, "Opus orchestrates, Sonnet builds", which left the coordinator free to write code itself.

Above both sits a single source of truth: the specs are authoritative (RUL-013). When code, a test, and a spec disagree, the spec wins — the specs/ requirement checklists and the @traces requirement ids a file claims to implement resolve design, contract, and version questions before anyone asks a human or trusts a test. Tests can enshrine the wrong behaviour; a spec is the contract. This is why the answer to a failing test is almost never "loosen the test."

Every substantial piece of work is tracked through the session CLI. A session begins with a purpose and objectives, logs work / artifacts / choices as it goes, records a handoff for the next session, and ends by updating context.md/context.json, regenerating CHANGELOG.md, and creating a pushed git tag. Nobody hand-edits context.json — it goes through the CLI, which does atomic, locked, self-healing writes. The point is that the project's memory is never a guess: what happened last session is recorded, and the next session resumes from it rather than re-discovering it.

Two rules bracket every commit, and both are mechanical gates now:

• Review before every commit (RUL-008): the pre-commit hook runs scripts/gate.py review-staged, the code review and the test review over the staged files, with the verdict derived in Prolog (Gate: review-staged). Any bug a review surfaces is fixed then and there, regardless of who wrote it. Since 2026-10-22 a worker meets the same gate before it may stop (Gate: commit-precheck).
• No AI attribution (RUL-007): Mohamed is the sole author of every commit, tag, and PR body. The commit-msg hook refuses an AI trailer or marker (Gate: ai-attribution).

Engineering rules — the non-negotiables

These are the invariants that make the codebase reviewable and the suite trustworthy. They are enforced by hooks, gates, and review tooling, not left to goodwill:

• No mocks, no fakes, no stubs (RUL-002). Real implementations only. Tests run against the real fleet: real Redis, real Memgraph, a real Prolog engine. The only boundary that may be swapped is the LLM. MagicMock, AsyncMock, patch, and any FakeX/StubX/DummyX class are banned and audited out (Gate: test-purity).
• No skips. No @pytest.mark.skip / skipif. Every test runs on every platform, every run. A failure is traced to root cause, never waved away as "flaky" or "pre-existing."
• No xdist (RUL-001). pytest -n auto / --dist deadlocks this suite (~78% stall) and is banned in four independent layers (a PreToolUse Bash guard, a pyproject.toml write guard, the code-review BANNED-DEP gate, and test-review rules T15/T16). Parallelism comes from the batched executor's per-worker process isolation, not from xdist.
• Absolute imports only — orchestrata_ultimate.*, never relative (Gate: prolog-gate).
• os.environ.get("X", default), never bare os.environ["X"]. Config is read at call time so CLI and test env overrides actually take effect; in tests, values are passed through an explicit injection seam — monkeypatch (including setenv/delenv) is forbidden suite-wide. Infrastructure ports and addresses take no default at all (RUL-011).
• Declare every dependency in pyproject.toml. No undeclared or manually-installed packages; the check-deps gate fails on a gap.
• Fix the code, not the test. When a test that asserts correct behaviour fails, the source is wrong and the source gets fixed — per the file's @traces reqs and its authoritative spec. Changing an assertion to match known-wrong output, weakening it to assert True/is not None, or deleting/skipping the test is the worst possible outcome, because it hides a real bug behind a green check.

The delivery flow

A change starts at an authoritative spec or a ruled ADR. The host Claude writes a brief; the dispatch gate refuses a brief without its five labels or its design authority; a worker builds it in its own worktree branched off origin/master; the worker cannot stop until the commit gate passes on its files; the commit runs the gate again; the push runs the affected tests; ship verifies and pushes; a deploy launcher installs and verifies the result. Each step is on the Workflow page.

Testing — one runner, real infrastructure, selected and capped

The canonical entry point is scripts/run_tests.py. It is not a thin wrapper around pytest; it is the whole pipeline (_main, read with the AST toolkit on 2026-10-23):

1. Freshness guard: six detect-and-refresh checks so no run consumes stale inputs (freshness-guard).
2. Affected resolution for --affected, before infrastructure: per-file verdicts, and a refusal that names the missing fact instead of a silent full run (RUL-040, affected-selection).
3. Infrastructure: Redis, Graph, Gateway and Kernel in parallel, then the Link check that the gateway really reaches the kernel over gRPC (suite preflight).
4. Five safety gates: source-provenance, parallel-safety, test-purity, prolog-boundary, rulings-lint.
5. Reuse: a proven report for the same code state, then the module-level green ledger (RUL-032, green-ledger); in gate mode the rehearsal-class modules are excluded (RUL-039, rehearsal-exclusion).
6. Test phase: a batched ThreadPoolExecutor on 8 workers, a selected-count line, a live progress line and a hard 300 s cap; then a solo retry of up to 25 batch-failed modules (test-phase-cap).

The pre-push hook runs it with --affected over the pushed commits; deploy/ship.sh runs it in its verify step. Recorded numbers from 2026-10-22: the promote's full run selected 21,295 tests in 1,080 modules, with 879 modules reused from the ledger and 16 rehearsal-class modules excluded (ship #14 promote log); an affected push ran 12 tests in one module in 31 s, with 6 modules (122 tests) reused (worked example). The figure this page used to give, 1,333 modules in about 10 minutes, dates from the session-242 speed campaign (the speedup story). CI and local runs both go through this script; scripts/validate_ci.py enforces that parity (validate-ci).

The Kernel (the Agent Runtime, gRPC on CXP_GRPC_PORT) is the beating heart: agents, CXP dispatch, governance, and the pipeline all route through it. It is marked required=True in start_test_services.py; if it does not start and pass a real gRPC verification, the run aborts. A suite run that skips the kernel is false-green — every kernel-dependent path silently no-ops or hits a fallback while the report shows green. There is exactly one sanctioned opt-out (--skip-services kernel), and any run that uses it is explicitly non-authoritative and must say so. The kernel must never be re-flipped to optional to make a suite pass; doing that re-introduces the false-green defect for everyone. (The gateway's PipelineKernelClient fallback is a production resilience path, not a licence to test without a live kernel.)

Two disciplines make the results trustworthy:

• Tests are observed, not waited on (RUL-009). You never run the suite in the foreground and read only the exit code. Runs stream per-module results through the live observer plugin (pytest_live_observer) and can be attached to with scripts/observe_pytest.py, so a killed run is still legible. The observer loads in every pytest a Claude session starts and refuses a multi-file session that does not come from the sanctioned runner (RUL-038 amendment).
• Solo-retry separates contention flakes from real breakage. A module that failed in a batch is re-run alone; if it passes it is logged to .flake_recoveries.ndjson as a load or order flake and treated as green. Over the cap of 25, failed modules are reported UNVERIFIED rather than judged. If you run a subset by hand and see a Memgraph-adapter failure, re-run that file alone before believing it.

Full detail lives on Running the test suite.

Coverage — the ≥90% per-file bar

Coverage is part of the Definition of Done (RUL-034), not a report you glance at afterwards. No code is "done" until it ships with real tests achieving ≥90% line coverage for every source file it creates or modifies — and the bar is per-file, not a project-wide average that lets a well-covered module hide a bare one. It is measured directly:

uv run pytest tests/<path> \
  --cov=<the.module.path> --cov-report=term-missing -o addopts=""

The -o addopts="" matters: it bypasses the default addopts so --cov runs single-process (this suite must never run under -n/--dist). In a checkout without UV_NO_EDITABLE=1, a plain uv run can re-sync the venv as an editable install, which the source-provenance gate then refuses at the next push; run the venv's python, or re-sync non-editable afterwards (source-provenance). The coverage config lives in [tool.coverage] in pyproject.toml, with generated protobuf (cxp/proto/) and examples already omitted.

The number has to be earned, not padded. Coverage must come from real assertions on real behaviour — assertion-free or filler tests written purely to move the percentage are rejected in review and do not count. When a file genuinely cannot reach 90% for a legitimate reason (a __main__ entrypoint, a real-browser driver, generated code), the residual is stated explicitly and quantified — it is never faked to hit the number. To plan the work, the AST toolkit maps what to test (branches/symbols/complexity) and seeds per-branch scaffolds (gen-tests); the scaffolds are then filled with genuine assertions, because a scaffold is not a test.

Deployment — the Incus fleets and the two graph databases

The services are Incus containers, not Docker and not localhost. docker ps and a local ss -tlnp show nothing, and that is expected; inspect the fleets with incus list. Nothing hardcodes an IP or a port: endpoints come from env/{dev,test,prod}.env, credentials from .env, so if a container IP changes you update the env file, never code. docs/LEVEL0.md (read 2026-10-23) gives three profiles on their own subnets and Incus projects, checked by the env-subnet-scheme gate:

tests/conftest.py pins the suite to env/test.env with a prod-subnet guard.

Do not conflate the two graph databases. Per docs/LEVEL0.md: the orchkg database is orchestrata's operational KG (sessions, gateway audit, CXP packets, code intelligence, specs, volks, ToolCall events), on a single Memgraph node in test and dev and on the HA orchgraph cluster in prod. The memgraph database is the shared semantic/RAG graph, written by both the knowledge-wiki project and orchestrata's semantic_graph ingest. The operational KG moved off Neo4j Community onto orchgraph; the record is KG migration to orchgraph. Treating one database as the other is the project's most common source of confusion. Policy runs in SWI-Prolog embedded in the kernel; the idle Prolog containers were retired from every fleet (ADR-0110).

The fleets are reproducible infrastructure, not hand-tended pets. OpenTofu owns every Incus environment; the live fleets were imported into a host-owned state on 2026-10-08 (f2cef12cb, ADR-0077). scripts/check_infra_drift.py runs tofu plan -detailed-exitcode and proves the bridge keys live; its last verdict, from 2026-10-22 19:01 UTC, is drift, and its hourly timer is disabled (fleet-drift). The rebuild rehearsal from an empty Incus host and the "exec, not cloud-init" build recipe are on Reproducibility & IaC.

Deployments are copies under /opt/<name> with their own venv and a recorded source commit, installed by launchers that verify identity and roll back on failure (ct-install, tools-deploy, hermes-deploy); see Deployment.

One deployment nuance worth knowing: the Prolog governance path runs on a vendored, patched janus-swi fork (scripts/_janus_patched/) so SWI-Prolog works under a genuinely GIL-off Python 3.14t build. It is wired as a path source built --no-build-isolation against this interpreter — uv sync --dev builds it; you never uv pip install stock janus. See Free-Threaded Prolog for the root cause and the fix.

Related

• Workflow
• Gates — master index
• Workflow improvements, Bug fixes
• Briefs, Dispatch ledger
• Tests, Running the test suite
• Deployment, Reproducibility & IaC
• Ruleset Engine — The Complete Picture
• Decisions, Rulings