gRPC Retirement — CXP moves to the cxpt direct transport

Branch: feat/cxp-direct-transport · Authorized: 2026-08-07 ("agreed merge, retire grpc, proceed as you recommend") · design of record: plans/grpc-retirement-design.md in the orchestrata-ultimate repo.

grpc.aio is gone from the CXP seam. cxpt — the vendored free-threaded direct transport — is now the only CXP transport, in prod and in every test. CXP_TRANSPORT=grpc is no longer a valid configuration; setting it raises at construction time instead of silently falling back to something else.

Why

Two independent drivers converged on the same removal:

1. SPEC-023 (GIL-off). grpc.aio's EventEngine threads re-enable the process-wide GIL, which violates the project's free-threaded doctrine. cxpt exists specifically to replace it, and as of 2026-08-07 prod runs on it (env/prod.env: CXP_TRANSPORT=cxpt, both units verified serving — see Measured results below).
2. The merge was blocked by it. MR !8's architecture-check CI job failed at uv sync --dev:

   failed to query metadata of file vendor/wheels/grpcio-1.82.1-cp314-cp314t-linux_x86_64.whl:
   No such file or directory
   

   The vendored free-threaded grpcio wheel lived under vendor/wheels/, which .gitignore excludes by design — the convention is "fresh clones rebuild it." CI has no wheel-build step, so any pinned [tool.uv.sources] path into vendor/wheels/ was an unresolvable dependency for a fresh clone. grpc was dead weight on the RPC path and an active CI blocker.

The key insight: protobuf and grpcio were never coupled

protobuf is the message codec (cxp_pb2.py, generated from cxp.proto). grpcio is the RPC runtime that used to carry those messages over HTTP/2. The two were separable all along — retiring gRPC meant the messages stayed and only the service stubs went.

The proof is CXPServicer's own contract: every RPC method's signature carried a context: grpc.aio.ServicerContext annotation, and that was grpc's only point of contact with the servicer. A grep 'context\.' over the original cxp/grpc_server.py returned zero hits — nothing in the servicer ever called a method on context. That single fact is why the carve-out was a move, not a rewrite: no behaviour changed hands. It also explains why direct_cxp_adapter.py had already been able to pass a bare sentinel object into CXPServicer in place of a real gRPC context — the cxpt path had effectively been running the "grpc" servicer in production all along, just without gRPC underneath it.

What moved, what was deleted

New/changed modules in src/orchestrata_ultimate/cxp/:

• servicer.py (new) — the transport-neutral protocol core, carved out of grpc_server.py: CXPServerConfig, CXPStats, CXPServicer, and the _env_*/_*_str_to_proto helpers. Every RPC method now types its second argument context: Any — it is whatever the calling transport passes, and nothing reads it. CXPServicer implements the six CXP RPCs (send_packet, send_packet_stream, health_check, get_status, receive_packet, subscribe, acknowledge) plus the proto⇄dict conversions either side of them.
• proto_codec.py (new) — CXPClientConfig (host/port/timeout/retries) and CXPPacketCodec, a stateless packet⇄protobuf converter. Both used to be private members of the retired grpc_client.py; DirectCXPClient had been reaching into GRPCClient._packet_to_proto/._proto_to_packet by constructing an unconnected GRPCClient purely to borrow them — a private-member dependency that only worked because GRPCClient.__init__ performed no I/O. CXPPacketCodec makes that a real, stateless API.
• transport.py — cxpt is now the only transport create_cxp_server/ create_cxp_client can construct. cxp_transport() still reads CXP_TRANSPORT, but only to reject it: a value of "grpc" raises ValueError naming the retirement and pointing at the design doc; any other value that isn't "cxpt" also raises. A retired transport must fail loudly at construction, not silently downgrade to whatever transport happens to be left.
• direct_cxp_adapter.py — DirectCXPServer/DirectGRPCClient now import from servicer.py/proto_codec.py instead of the deleted grpc_* modules. They keep the Direct/GRPC naming for now, purely to avoid churning call sites in the same change that removed the transport they used to mirror.

Deleted outright: cxp/grpc_server.py, cxp/grpc_client.py, cxp/proto/cxp_pb2_grpc.py, scripts/build_ft_grpcio.py, scripts/build_ft_grpcio_tools.py, vendor/grpcio-ft/, vendor/grpcio-tools-ft/. Confirmed at HEAD: none of these paths exist, and grep -rn '^import grpc\|^from grpc' src/ tests/ returns zero hits.

grpcio and grpcio-tools are now entries in .claude/banned-libraries.txt, with the retirement rationale inline in the file so a future uv add grpcio gets caught by the no-banned-deps gate instead of silently reintroducing both the GIL violation and the unresolvable-dependency class.

Eleven call sites were rewired onto create_cxp_server/create_cxp_client or the Direct* classes directly: agents/runtime_main.py, cxp/transport.py (×2), cxp/direct_cxp_adapter.py (×2), kernel/cli_main.py, kernel/dispatcher.py, knowledge_graph/server.py, gateway/server.py, cxp/proto/__init__.py, scripts/start_test_services.py.

Six grpc-specific test modules were retired or repointed: tests/cxp/test_grpc.py, test_grpc_client.py, test_grpc_fields.py, test_grpc_kernel_endpoint.py, test_proto_grpc.py were deleted; tests/gateway/test_kernel_reply_consumer.py was repointed at the cxpt path rather than deleted (it still exists at HEAD).

Two production defects found and fixed during the cutover

Cutting cxpt over from "the alternative transport" to "the only transport" made two pre-existing gaps load-bearing. Both were found and fixed in the same change, not deferred.

1. Unary calls had no timeout. AsyncDirectClient.call() (cxp/direct_transport.py) awaited its reply future with no deadline. A peer that died — or simply never answered — after a request was submitted left that future pending forever; there is no connection-level event that completes it. Measured: a gateway talking to a kernel that had gone away blocked until pytest-timeout killed it at 120s, and would have blocked indefinitely in production. Fixed with _await_reply(), which wraps the future in asyncio.wait_for(fut, timeout=timeout) whenever a timeout is configured.

2. The client's retry loop never applied the configured timeout — timeout_seconds and max_retries were dead configuration. CXPClientConfig.timeout_seconds was read by nothing, and until this change DirectGRPCClient.send_packet() called the transport once and never retried, so max_retries (default 3) was also never honored. Every caller constructing a client expecting three attempts was silently getting one, turning every transient blip into a propagated failure. Fixed via _call_bounded(), which threads self._config.timeout_seconds through client.call(..., timeout=...), and a real retry loop in send_packet() that honors max_retries with escalating backoff (0.5 * (attempt + 1) seconds), matching what the retired grpc client's callers already assumed. max_retries=0 is honored as "make zero attempts," not "retry forever."

Both defects predate the cutover — they existed in direct_cxp_adapter.py and direct_transport.py before 2026-08-07 — but only became load-bearing once cxpt stopped being an alternate path and became the transport.

Behavioural difference worth recording: grpc.aio's insecure_channel() connects lazily — construction never fails, and a dead peer only surfaces on the first RPC. cxpt's AsyncDirectClient dials a real TCP (or Unix-domain) socket at connect() and raises OSError immediately if nothing is listening. Callers that assumed lazy-connect semantics from the grpc era need to expect that connect() itself can fail now, not just the first call().

Also worth recording: a raising handler on the servicer side (an exception in _packet_handler) produces a CXPReceipt(success=False, ...) — that is a successful round trip carrying a business-logic failure, and send_packet()'s retry loop correctly does not retry it. Retry is for transport failure (RpcError, TimeoutError, OSError) only; a successfully-delivered failure receipt is not one of those.

protoc / gencode contract

grpcio-tools regenerates cxp_pb2.py/cxp_pb2.pyi from cxp.proto (scripts/gen_cxp.py), and tests/scripts/test_gen_cxp_cli.py drives it through a real protoc subprocess (no mocks). So the tool couldn't just be dropped — but it is now deliberately not a project dependency.

Two independent reasons:

• No cp314t wheel. Checked 2026-08-07: grpcio-tools 1.83.0 publishes cp314 wheels but no cp314t. Depending on it as a project dependency meant vendoring a hand-built free-threaded wheel under the gitignored vendor/wheels/ — the exact unresolvable-dependency class that broke MR !8's CI.
• It doesn't need to be free-threaded. protoc is a code generator invoked as a subprocess; it emits text and exits. Nothing about the GIL in that child process touches this project's runtime.

gen_cxp.py now fetches grpcio-tools on demand into a throwaway, GIL-enabled environment rather than depending on it at all:

uv run --isolated --no-project --python /usr/bin/python3.14 \
    --with grpcio-tools==1.80.0 \
    python -m grpc_tools.protoc ...

uv caches the wheels, so repeat runs are fast and offline. The version is pinned, and the pin is load-bearing: protoc's output carries a gencode version that the installed protobuf runtime validates at import, and a runtime older than the gencode is a hard VersionError. Measured 2026-08-07: the system /usr/bin/protoc 35.1 emits gencode 7.35.1 against this project's protobuf>=4.25 runtime and fails at import; grpcio-tools==1.80.0 emits gencode 6.31.1, which that runtime accepts. Bumping the pin means re-verifying that the regenerated cxp_pb2.py still imports.

cxp_pb2_grpc.py is no longer generated at all — there is no gRPC service to generate stubs for.

Measured production results

Cutover verified 2026-08-07: env/prod.env set CXP_TRANSPORT=cxpt, both units restarted together (kernel pid 1377133, gateway pid 1377190).

• Kernel listening on 0.0.0.0:50053; gateway on 8765/8766; cxpt mapped.
• health_check over cxpt: 0.76ms.
• get_status over cxpt: 1.02ms.
• Authenticated gateway.ping: 4.0ms.
• Authenticated volk.list: 3.2ms.
• Kernel file descriptors: flat at 20 (no leak under the new transport).

Revert path while grpc_server.py/grpc_client.py still existed: delete CXP_TRANSPORT=cxpt from env/prod.env and restart both units. After the servicer extraction landed there is no grpc transport left to revert to — reverting now means git revert of the retirement commit(s) plus rebuilding the vendored wheels.

Known open: the CXP WebSocket transport has no route to it

cxp/ws_client.py and cxp/ws_server.py (WSServer, WSServerConfig) have zero import sites anywhere else in src/. Verified this was not caused by the gRPC retirement: at HEAD, ws_server.py was already the only file in the tree defining WSServer, and neither deleted grpc module (grpc_server.py, grpc_client.py) ever referenced either WS module. This predates the retirement.

What the retirement did change is that cxp_transport() now recognizes exactly one transport (cxpt) and raises on anything else — so the WS modules are a complete, unreachable transport implementation with no selection path into it at all, not even a broken one. Decision pending: wire ws as a third selectable value in cxp_transport(), or delete the two modules. Not resolved as part of this retirement.

References

• plans/grpc-retirement-design.md (orchestrata-ultimate repo) — design of record, order of work, verification checklist.
• .claude/banned-libraries.txt — grpcio/grpcio-tools ban + rationale.
• src/orchestrata_ultimate/cxp/servicer.py, src/orchestrata_ultimate/cxp/proto_codec.py, src/orchestrata_ultimate/cxp/transport.py, src/orchestrata_ultimate/cxp/direct_cxp_adapter.py, src/orchestrata_ultimate/cxp/direct_transport.py, scripts/gen_cxp.py.
• Kernel fd leak (cxpt) and the env-ownership marker — a separate defect found the same day, in the cxpt native transport that this retirement now depends on exclusively.