Suite parallelism — measured floor and rejected levers (2026-08-05)

Authoritative source: reports/claude-failure-report-2026-08-05.md in the repo (session-260 addendum + final ledger) and commit 76fd6431.

The floor

Winning config -j 14 --batch-multiplier 2: clean-green band 251–265s for 30,668 tests / 1523 modules. Arithmetic: 2437s suite test time ÷ 14 workers = 174s + 2 × ~37s per-bin overhead (imports + collection

• fixture setup) ≈ 250s. Box: i7-8850H, 6 physical cores / 12 threads.

Levers measured and their verdicts

Standing traps this session codified

• Red runs poison the duration cache — the executor now folds observed durations only from green modules.
• Worker-band databases are provisioned, not sediment — run_tests.py passes --ensure-worker-dbs=<j> to preflight; purging orchkg_w* is now always safe.
• Ambient FORCE_COLOR breaks machine-parsed child output — the five parsing tools force NO_COLOR for children; launch gates with env -u FORCE_COLOR anyway.
• Kernel accept queue wedges permanently after backlog overflow (129/128) — restart orchestrata-kernel in the app container; liveness fix is backlogged.

To get under 240s

Shrink the work, not the schedule: per-bin import/collection cost (~37s/bin) and fixture time on the hottest modules. Or run on more cores — the --backfill seam (default off, tested) is designed for that hardware.

Addendum (2026-08-05, session 260 late): two root causes behind the numbers

Suite-total undercount on all-green runs. Batched-executor totals fold from batch JUnit only; a killed batch has no JUnit, so its COMPLETED modules were live-log-rescued green while contributing zero tests to the totals (iter-19: expected 30671, measured 30432). Fixed: the live-log reducer reconstructs per-file terminal-outcome counts, the rescue carries them into totals, and the LAST active module of a killed batch is no longer rescued green on partial evidence (killed-between-tests is indistinguishable from completion) — it takes the Phase-2B retry. reports/run-tests-latest.json now persists per-module executed-test counts every run.

Kernel accept-queue wedge (129/128) retry driver. claude_cli_adapter fails with "Claude CLI timed out after 600s"; "timed out" is a transient marker, so the llm client's 6-attempt retry loop pinned one kernel handler for up to ~62 min per volk call. Fixed with a total-elapsed retry budget (LLM_RETRY_TOTAL_BUDGET_SECONDS, default 900s). Commits fbfaf091 + d2ed99a2 on feat/cxp-direct-transport.

Acceptor-liveness closure (same day, later): the wedge-PERMANENCE half is also fixed at the C level — cxpt's acceptor_main exited its thread on any non-EINTR accept() failure, so one EMFILE burst killed accepting forever. Transient errnos now retry with 50ms backoff; a failed reader pthread_create closes the connection instead of leaking the fd. Red-green proven against real EMFILE (pre-fix binary FAILS the new liveness test, fixed binary PASSES an application-level echo). Commit 887066e4; deployed to the app container (kernel gRPC + gateway probes green post-restart). Both wedge halves are now closed: the llm retry budget bounds handler occupancy; the acceptor survives the pressure spike.

Addendum (2026-08-12, session 262): parallelism floor assumes per-worker isolation the Redis fixture didn't actually have

The whole "per-worker process isolation, not xdist" model this page's floor is measured against (per-worker Neo4j via NEO4J_URI_GW<N>, per-slot Redis DB) depends on every test's Redis writes actually landing in an isolated namespace. tests/conftest.py::redis_client does not: it connects with no db= argument, landing on logical database 0 — the same database REDIS_DB=0 in env/test.env names for the live test-fleet kernel and gateway. Teardown scopes only to f"{test_id}:*". An unscoped key from that fixture is a live cross-worker collision on the shared floor this whole page is about, not an isolated-worker cost.

scripts/gate_parallel_safety.py Gate 0.4 (Redis key namespacing) is the static check for this — rebuilt this session from an AST taint model that trusted any function call receiving a test_id-derived argument (proof nothing about what the call returned) to a model that trusts only str.format(...), plus a corrected receiver/verb inventory (flushdb had measured 0% detection before — 5 live sites). Rebuild found 130 real unscoped-key sites across 51 files, all now either rewritten to derive from test_id or waived with a reasoned # gate-ok:. Full writeup: a checker must prove it looked; day-to-day guidance: running the test suite, §6.