Knowledge Wiki Orchestrata Ultimate

Home / Infrastructure / Kernel

Kernel Service

The kernel is the beating heart of orchestrata. If it is not run and tested, the project fails. It is REQUIRED infrastructure, never optional (docs/kernel-is-required.md, AGENTS.md Hard Rule 7).

The kernel is an out-of-process service started as python -m orchestrata_ultimate.agents.runtime_main. It dispatches agent intents, and it is the central coordination process for agents, CXP dispatch, governance and the pipeline. It serves CXP over the cxpt transport (CXP_TRANSPORT=cxpt) on CXP_GRPC_PORT, which is 50053 in every env profile. gRPC was retired on 2026-08-07 (commit 11ebfc2e7). The GRPC in the variable names and the "gRPC server" module docstring are left over from before that.

Where it runs (measured 2026-10-23)

Copy Where State Deployed commit
prod host unit orchestrata-kernel.service, from /opt/orchestrata, CXP_GRPC_HOST=localhost disabled, inactive: DeepChat disabled it on 2026-10-21 4ac0db570
test orchestrata-kernel in orchestrata-test:app, CXP_GRPC_HOST=10.53.2.13 active 22363abb2 (2026-10-12), 546 commits behind
dev env/dev.env points at localhost:50053 no unit; started by hand when needed -

The prod unit's ExecStartPre waits up to 215 s for the graph and creates /var/lib/orchestrata/session-root. Section 3 of Host services records why it is a unit: "a hand-started kernel keeps whatever .env it was launched with".

Diagram infrastructure-kernel-3

REQUIRED: never skip it in a test run

A test-suite run that does not start AND verify the kernel is not a valid run: kernel-dependent paths go silently untested and the suite reports false-green.

scripts/start_test_services.py marks the kernel required=True in its SERVICES table, next to redis, graph and gateway. This was read through the AST toolkit in wt-MAP-PRECOMMIT on 2026-10-23. That means every run:

  1. starts the kernel if it is down (start_kernel), and
  2. verifies it by opening and closing a real CXP client connection (verify_kernel, through create_cxp_client).

After those four services settle, run_all checks the gateway-kernel link (check_kernel_link); the link is also required. If any required check fails, run_all().all_required_ok is False, start_test_services exits non-zero, and scripts/run_tests.py aborts. No kernel means the run fails, by design.

A gap in verify_kernel. If importing the CXP modules raises ImportError, verify_kernel returns status running with the detail "package not installed (skipped)". run_all counts running as OK, so that case passes the required check without a kernel connection. The failure detail also still says "gRPC connect failed".

The required start-and-verify flow:

Diagram infrastructure-kernel-4

The defect this replaced

The kernel used to be required=False, which made _run_one short-circuit it to skipped — optional service not started without ever starting it. Every full-suite report showed ⏭️ Kernel … skipped even though CXP_GRPC_PORT was set the whole time. It was mis-classified as optional; flipping it to required=True is the fix.

The only sanctioned opt-out

A rare environment that genuinely cannot run the kernel may skip it loudly and explicitly:

python scripts/run_tests.py --skip-services kernel
# or: python scripts/start_test_services.py --skip kernel

Such a run is non-authoritative — state that in any report. Never re-flip the kernel to optional to make a suite pass.

Connection

CXP_GRPC_HOST:CXP_GRPC_PORT   # from the env profile, read at call time, never hard-coded
                              # prod localhost:50053, test 10.53.2.13:50053, dev localhost:50053

Status and start

python scripts/start_test_services.py --status   # check only (never fails)
python scripts/start_test_services.py            # check, start, verify all 4 services, then the link

Production fallback (not a test opt-out)

At runtime, if the kernel endpoint is unreachable, the gateway logs a warning and falls back to PipelineKernelClient (AGENTS.md Hard Rule 4). That is a production resilience path. It is NOT a licence to run the test suite without a live kernel. The prod gateway unit declares Wants=orchestrata-kernel.service, so starting the gateway also starts the kernel, even though the kernel unit is disabled. Which functions are lost in the fallback has not been traced.