Orchestrata Ultimate › Plans › parallel-flakiness-fix-plan

Parallel Test Flakiness — Root-Cause Fix Plan

Source: plans/parallel-flakiness-fix-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-06-18 · 159 lines.

Prose pending.

Plan text

Parallel Test Flakiness — Root-Cause Fix Plan

Date: 2026-06-18 · Status: proposed (review before applying) · Author: Mohamed

Grounded in a 4-agent AST+code analysis of the runners, fixtures, src/, and tests. Every claim below is file:line-verified or empirically confirmed.

0. The reframe (why this is tractable and fast to verify)

Two confirmed facts change everything about how we fix and verify this:

1. The runners run one subprocess per test module (core/threading/test_executor.py:68,133 — [sys.executable,"-m","pytest",path,...] dispatched by a ThreadPoolExecutor; confirmed: distinct PIDs per module). Modules do not share a Python process. Therefore cross-module flakiness is shared-backend collision (one Redis, one Neo4j), not in-process data races. In-process/no-GIL races (Tier B below) are real production bugs but are not the dominant cause of parallel test flakiness.

2. Test order is deterministic — pytest-randomly is not installed (confirmed). The -p no:randomly flags in our scripts are no-ops. So flakiness is not order-driven; it is 100% concurrency/shared-backend. That means every flake is structurally reproducible — we never need probabilistic "run it 5×30min" hunting.

Decisive infra constraint: Neo4j is Community 5.26 (confirmed) → a single user database; per-worker Neo4j DB isolation is impossible. Redis does get per-worker DBs (PYTEST_XDIST_WORKER=gw{slot} IS injected by both runners → _worker_redis_db() → db=slot; confirmed). So the fix must make Neo4j isolation rest entirely on token-scoped writes + token-scoped teardown, and must eliminate every global-wipe.

Verification philosophy (the user's core ask): convert nondeterministic collisions into deterministic, single-run, seconds-fast signals via (a) an isolation-assertion conftest hook, (b) static AST gates, (c) targeted concurrent micro-stress of the named colliding module sets. No 30-minute runs anywhere.

1. Root causes, ranked

Tier A — cross-module backend corruption (THE flakiness)

Tier B — no-GIL / event-loop correctness (production-real; flakes only within a module)

Tier C — hygiene

• governance/escalation_handler/{handler,sla,resolver}.py, kernel/audit/writers.py: asyncio.get_event_loop() inside running coroutines → use get_running_loop().
• tests/knowledge_graph/conftest.py:28,38: dead session-scoped neo4j_async_driver fixture with unawaited driver.close() → make function-scoped+async or delete.

Dismissed (not bugs): the except A, B: "syntax errors" several agents flagged (kernel/event_bus.py:326, kernel/audit/logger.py) are valid on 3.14 (PEP 758) — the full package already imports with 0 failures. No action.

2. Fix phases (each independently shippable + independently verifiable)

Phase 1 — Stop the corruption (Tier A1, A2, A5) — fixes the known flakes

• A1: re-scope test_kg_sync_extended.py:123-127 to {project_id:$pid}/{test_id:$tid} exactly like lines 121-122. No bare-label or fixed-id deletes.
• A2: scope the MAPS_TO sweep to the test's class ids (already has $s/$t available at :235) — delete only this test's edges.
• A5: replace every hardcoded db=N with db=_worker_redis_db(); remove flushdb(); switch to token-scoped scan_iter(match=f"...*{test_id}*") teardown (the GOOD baseline idiom from the three regression tests).

Phase 2 — Make it un-recurrable (enforcement; this is also the verifier — see §3)

• 2a. Isolation-assertion autouse conftest hook (§3a).
• 2b. Static AST gates for the anti-patterns (§3b) wired into the review engine + a standalone CI script.

Phase 3 — Honor not_parallelizable (A4)

• Teach run_tests_streaming.py + test_executor.py to read each module for the marker (cheap: grep the file, or pytest --collect-only -m not_parallelizable) and run those modules serially after the parallel batch (or in a dedicated -j 1 lane). Removes a whole class of collisions for modules that genuinely can't be scoped.

Phase 4 — A3 SUT scoping + A6/A7

• A3: add an optional project/test_id scope parameter to _build_monorepo_context / _query_project_tree (default preserves prod behavior); the two architect tests pass it. This is the only SUT change; review carefully.
• A6: replace os.chdir with passing an explicit root/cwd to the SUT (the two assertion-bearing tests).
• A7: narrow _cleanup_test_prefix to the running test's test_id (it already has access); confirm Scheme-B fixtures only ever write test_id-prefixed keys.

Phase 5 — No-GIL + loop affinity (Tier B, C)

• B1, B2, B3 first (highest production value, simplest). Then B4. Then B5/B6 (per-loop rebind in ProviderManager/RedisStateManager + drop _loop caching) and the Tier-C get_running_loop() sweep.
• These prevent flakiness/incorrectness under future in-process threading and under real no-GIL production load.

3. The fast-verification harness (built once, in Phase 2; verifies all phases)

Goal: every fix is confirmed in seconds, deterministically. Total budget for the whole plan's verification: < 3 minutes, no full-suite run.

3a. Isolation-assertion conftest hook — turns collisions into deterministic single-test failures

Add an autouse, function-scoped hook in tests/conftest.py that, around each test touching a backend:

• Neo4j sentinel: before the session's parallel batch, a sentinel node (:_Sentinel {id:'_SENTINEL_GLOBAL_'}) is seeded. After each test, assert the sentinel still exists and that core labels weren't globally wiped (e.g. count of :_Sentinel unchanged). A global-wipe teardown (A1/A2) deletes/أffects it → the offending test fails immediately, by name, in a normal run.
• Residue check: after each test, assert there are no Neo4j nodes created during the test that lack a test_id/token property, and no Redis keys on this worker's db lacking the test_id prefix (scan a bounded sample). Unscoped writes (A3/A7) fail deterministically.
• Gate behind an env flag (ISOLATION_ASSERT=1) so it's a verification mode, not always-on overhead.

→ One ISOLATION_ASSERT=1 run over the suspect dirs names every offender. No repetition needed.

3b. Static AST gates — instant, no test run; also prevents recurrence

New checker check_test_isolation(filepath, tree) (mirror the existing check_env_access pattern in verification/review/checks_safety.py, wired into engine.py, documented as a standards.py rule) flagging, in tests/:

• flushdb( / flushall( calls;
• DETACH DELETE/DELETE over a bare label or fixed-literal id (no $param/test_id in the MATCH) — AST-match the Cypher string args;
• hardcoded db=<int-literal> in redis client construction;
• os.chdir( in a test module. Plus a standalone scripts/check_test_isolation.py for a millisecond pre-commit/CI gate. (Directly satisfies the earlier "add detection so these never go undetected" goal.)

3c. Targeted concurrent micro-stress — deterministic repro of the named collisions

scripts/stress_parallel.sh <modules...> [rounds] runs a chosen module set concurrently under the streaming runner N times, stops on first failure:

for i in $(seq 1 "${ROUNDS:-20}"); do
  python scripts/run_tests_streaming.py "$@" -j 4 -t 120 || { echo "FLAKE round $i"; exit 1; }
done

The colliding sets are already identified, so this reproduces in seconds:

# A1/A3 repro (architect-empty + Action-wipe collide):
ROUNDS=20 scripts/stress_parallel.sh \
  tests/session/test_kg_sync_extended.py \
  tests/agents/volks/architect/test_context_mixin.py \
  tests/knowledge_graph/test_kg_sync.py
# A5 repro (db=15 flushdb cluster):
ROUNDS=20 scripts/stress_parallel.sh \
  tests/providers/test_cost_tracker.py \
  tests/kernel/test_audit_logger.py \
  tests/core/test_redis_factory.py

Before fix: fails within a few rounds. After fix: 20/20 green in well under a minute.

3d. Keep -j ≤ 16 always (Redis has 16 DBs; slot 16 → "DB index out of range"). Document in the runners (or clamp).

4. Per-fix → fast deterministic check

5. Sequencing, effort, risk

1. Phase 2b (static gates) first — instant, read-only, and it enumerates the full offender list to size Phases 1/4 precisely. ~½ day. Risk: none.
2. Phase 2a (isolation hook) — the deterministic detector. ~½ day. Risk: low (env-gated).
3. Phase 1 (A1/A2/A5) — the actual flake fixes. ~½ day. Verify with 3c. Risk: low (test-only).
4. Phase 3 (honor marker) — ~¼ day. Risk: low (runner-only).
5. Phase 4 (A3 SUT scope, A6, A7) — ~½ day. A3 is the one SUT change → review before applying. Risk: low-med.
6. Phase 5 (Tier B/C) — ~1 day. Independent of the above; high production value. Risk: low (additive locks / per-loop rebind), verify each with a focused thread/loop stress.

Whole-plan acceptance: static gate exits 0; ISOLATION_ASSERT=1 over tests/knowledge_graph tests/session tests/providers is clean; all 3c stress sets 20/20 green; one normal -j 8 streaming run of those dirs green. All of that is < 3 minutes total, deterministic — never a 30-minute probabilistic run.

6. Open decisions for review

• A3 SUT change: add an optional scope arg to the architect context queries (default = current prod behavior) vs. make the two tests assert tolerantly. Recommend the scope arg (correct + reusable).
• Phase 3 mechanism: serial lane after the parallel batch (simplest) vs. a separate -j1 invocation. Recommend serial-after.
• Scope: do Tier B/C now (Phase 5) or defer? They don't cause the current cross-module flakiness but are real no-GIL bugs. Recommend doing B1/B2/B3 now (cheap, high value), defer B4-B6/C if time-boxed.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks