Kernel fd leak (cxpt) and the env-ownership marker

Two defects found on 2026-08-07. Both had been misread as environmental — "Memgraph contention", "load flakes" — across several full-suite runs. Neither was environmental. Both were deterministic, and both were ours.

The shared lesson: a failure whose message names nothing gets attributed to the nearest plausible noise source. "0 rows", "record is None", "timeout in teardown" name no subsystem, so they were charged to infrastructure. The tell in both cases was reproducibility — the victim passed alone, every time, and failed in company, every time. Load is not that obedient.

1. The kernel fd leak (cxpt native transport)

Symptom

scripts/start_test_services.py --status reported the kernel down:

❌ Kernel  port=50053  failed  not listening on 10.53.2.13:50053

while systemd reported it healthy — active (running) since ... 1 day 2h ago, main PID alive. Gateway WebSocket tests failed in teardown on a 120s pytest-timeout, and a full gate produced 4 red modules plus 57 modules with no result at all.

Diagnosis

open fds: 1024
Max open files  1024
ss -tanp | grep pid=<kernel>
   932 CLOSE-WAIT 10.53.2.1     <- the host
    39 ESTAB      10.53.2.11    <- redis
     3 ESTAB      10.53.2.10    <- graph
     1 LISTEN

CLOSE-WAIT is unambiguous: the peer sent FIN and the local side never called close(). 932 of them, against a 1024 soft limit, so accept(2) returned EMFILE forever. The process was up, listening in systemd's view, and unreachable in fact.

The suite's CXP transport is cxpt (env/test.env: CXP_TRANSPORT=cxpt) — our own C extension, not grpc.aio. In scripts/_cxpt/cxpt/_native.c, conn_reader broke out of its read loop on peer disconnect, set cn->alive = 0, and returned. Connection fds were released only in Server_dealloc — i.e. when the entire server was destroyed. A server that stays up (the kernel runs for days) therefore leaked one fd per client that ever connected.

Fix

Close in conn_reader, under cn->wlock, setting fd = -1:

pthread_mutex_lock(&cn->wlock);
if (cn->fd >= 0) { shutdown(cn->fd, SHUT_RDWR); close(cn->fd); cn->fd = -1; }
pthread_mutex_unlock(&cn->wlock);

Two subtleties that are load-bearing:

• Server_reply must re-check fd >= 0 under the same lock. It tests cn->alive before taking wlock; a writer that passed that test and then blocked on the lock would otherwise write into a reused fd number — a reply delivered to a different peer's socket. Far worse than the leak.
• The Conn struct is deliberately NOT freed here. Server_reply reads s->conns[id] under conns_lock and uses the pointer after dropping it, so freeing in the reader would be a use-after-free. Server_dealloc still owns the structs.

Verification

Pinned by tests/cxp/test_direct_transport_fd_release.py, which also asserts the server still serves after heavy churn — the property the locking above protects.

2. The uv build-cache trap (cost a wrong measurement)

While fixing the above, a measurement said 40 fds still leaked against what was actually the pre-fix binary. A uv run had re-synced cxpt from uv's cache.

uv keys its build cache on (name, version), and these vendored extensions were pinned at versions that never change. So editing a .c file and reinstalling can serve a previously-built wheel and copy a stale .so into the venv with a fresh mtime.

scripts/check_native_build_freshness.py cannot catch this — it compares mtimes, and the restored file's mtime is newer than the source. The same trap had already cost a session on 2026-08-05 (Server_dealloc segfault: the fix looked wrong; it was not installed).

Fix — remove the failure mode instead of detecting it. cxpt, ctok and codemod_fast now derive their version from a sha256 of their own .c/.h content:

cxpt==0.1.0+1e108137d506

Changed source → different version → different cache key → the cache cannot serve the old wheel. cxpt>=0.1.0 still matches, since PEP 440 ignores the local segment. Proved by editing a .c and reinstalling with the cache enabled and no build-dir wipe: the new code shipped.

To confirm a binary really contains your edit, plant a unique string in the .c and strings the installed .so. That is what settled it.

3. The env-ownership marker

core.env_bootstrap.load_env_contract applied env/<profile>.env with unconditional override. Any of ~31 test modules that reaches an entrypoint ran it mid-process and clobbered per-process specializations for the rest of the batch:

KG_GRPC_PORT   50161      -> 50052
KG_DATABASE    orchkg_w9  -> orchkg
MEMGRAPH_URIS  ""         -> bolt://10.53.2.10:7687

That splits the brain: production resolves KG_DATABASE at call time and got orchkg; each test froze tests.infrastructure.KG_DATABASE at import and kept orchkg_w9. Writes landed in one database, read-backs looked in the other, and the failures said only "record is None" / "0 rows". Four authoritative gate runs were charged to Memgraph contention.

Proof, not inference: test_audit_logger_enforcement_proof alone → 2 passed in 1.01s. Preceded by test_server.py in one process → 3 failed in 211.00s, text byte-identical to the gate's .flake_recoveries.ndjson. After the fix, the same reproduction → 7 passed in 1.01s.

Why a marker, not a key list

scripts/start_test_services.py already carried a hardcoded three-key exclusion; load_env_contract carried none. Two loaders, two policies.

A name list cannot work, because the same KG_DATABASE may arrive from conftest (authoritative — must win) or from a stale shell export (must lose), and the key name cannot tell them apart. Only the process that computed the value knows. This machine really does carry such exports: tests/config.py aliases GRAPH_USERNAME/GRAPH_PASSWORD precisely because the shell still exports the NEO4J_* credentials retired on 2026-07-27.

So the claim travels in the environment — ORCHESTRATA_ENV_OWNED_KEYS, set via declare_owned() — and both loaders consult owned_keys(). Membership, not truthiness: a claimed key may be deliberately blank (conftest empties MEMGRAPH_URIS so the standalone test node cannot HA-discover its way to the prod coordinators).

scripts/validate_ci.py gate 13 holds both loaders to it via an AST check that each overlay function calls the marker, plus a ban on reintroducing a private key list.

The first draft of that gate matched the bare name as a substring and stayed green after the call was removed — owned_keys still appears in a docstring and in declare_owned. Mutation-test your gates: a gate that cannot fail on its primary case is not a mechanism.

4. Diagnosing order-dependent failures

A module that passes alone and fails in a batch was broken by something else in its process — so the batch peers are the diagnosis. The flake log recorded only the victim and its failure text, and the junit temp dirs that could reconstruct the batch are deleted when the run ends. One real order-dependent failure (test_check_native_build_freshness, gate 10) was already unrecoverable minutes later.

.flake_recoveries.ndjson records now carry batch_peers.

That change came with its own trap, worth repeating: consumers tally modules by iterating the results map, so any non-module key left in it reads as an extra FAILED module. Adding __batch_plan__ broke scripts/run_tests_threadpool.py (which popped only the two keys it knew) and the executor's own tests. Fixed with AGGREGATE_KEYS + pop_aggregates() beside the key definitions, so no caller names keys by hand.

Related

• [[free-threaded-prolog]] — the other native-extension load-bearer
• [[decisions]] · [[blockers]]