Orchestrata Ultimate › Plans › claude-only-tests-handoff
Claude-only test campaign — handoff (2026-08-04)
Source: plans/claude-only-tests-handoff.md at 8a2cb42ad · last changed 2026-08-05 · first committed 2026-08-05 · 62 lines.
Prose pending.
Plan text
Claude-only test campaign — handoff (2026-08-04)
Policy (Mohamed): tests exercise Claude ONLY; never OpenRouter/qwen/minimax/deepseek. Follow-ups: "copy claude's answer as if it is the same for the other llms" and "do not call the llm in the tests, just return any viable text and save the time". Runs must be threaded AND observed (never blind buffered pytest).
DONE (in working tree, uncommitted, gate-clean where noted)
claude_only_llmfixture (tests/conftest.py): pinsSPIKE_PROFILE=claude_cli(in-procselect_profile+ env for subprocess),LLM_CASSETTE_DIR,PIPELINE_RETRY_BACKOFF_S=0,LLM_CASSETTE_SYNTH=1. Restores all on teardown.llm_replayfixture: now also setsLLM_CASSETTE_SYNTH=1.- autouse
_synth_llm_for_integration_lane(tests/conftest.py): everyllm_integration-marked test gets cassette dir + synth + backoff=0. Scoped by marker only; default lane untouched. claude_cliprofile fix (configs/spike-profiles.yaml): added adefault: {provider: claude_code, model: sonnet}sentinel — it was unusable for full pipelines (missing volk aliases → LLMConfigError at stage 1).- content_key mechanism (
llm/_cassette.py+llm/client.py): one Claude recording serves every model (hash msgs+temp, not model); on exact miss, replay Claude's answer under the requesting model's identity. RECORD saves both keys. - synth mode (
llm/client.py):LLM_CASSETTE_SYNTH=1→ canned viable text (_SYNTHETIC_LLM_TEXT, prose+code block) on a cassette miss, no call, no raise. - Poles collapsed + verified (ruff+pyright 0):
test_ultimatetui::handle_intent_success36.5→0.37s;test_pipeline_runner_generatedsmoke 35.9→0.39s;test_e2e_runx2 (300/180s→<0.5s);test_pipeline_quick_test_cli+test_test_pipeline_smokesubprocess (300s→~1.2s);test_pipeline_runner_checkpoint(1) Claude-pinned. - Builder-fixed real bugs:
e2e_run._setup_envdeepseek pin (needs--profile);ModuleNotFoundError: scriptsin 2 CLI scripts (PYTHONPATH fix in-test). - Builder 5 (stopped mid-verify at 5-min cap, edits ruff-clean): wired
test_qwen_minimax_e2e.py(7 tests,llm_replay),test_llm_profiles_integration.py,test_deepseek_coder_driver.py,test_evaluator_llm_judge.py; recorded 18 cassettes. tests/llm/test_cassette.py: 15 tests pass (content_key + synth + record-both).- Hook
.claude/hooks/route-pytest-to-threads.sh: rewritten to route multi-module pytest →run_tests_streaming.py … -a="-p" -a="…pytest_live_observer"+ preserve-m/-k+ point atobserve_pytest.py. All 6 paths verified, shellcheck clean. Loads next session (hooks load at session start — that's why it didn't fire this run). - Tasks #32/#33/#34 complete.
REMAINING TODO
- Run the llm_integration lane OBSERVABLY (hook now enforces it):
python scripts/run_tests_streaming.py <28 marker files> -j 8 -a="-m" -a="llm_integration" -a="-p" -a="orchestrata_ultimate.core.threading.pytest_live_observer"backgrounded, thenobserve_pytest.py list/attach. File list:grep -rlE llm_integration tests/ --include='*.py'(drop conftest.py). - Neutralize real-subprocess LLM tests synth can't reach (they don't go through
call_llm/_execute_with_retry): chieflytests/drivers/test_claude_code_driver_live.py(spawns the realclaudebinary). Options: a stubclaudeon PATH returning viable text, or skip-guard, or route the driver through the synth seam. Audit the full 28-file set for other direct-provider/subprocess callers. - Confirm the lane is green + fast end to end under synth; capture per-module timing.
- Full gate on every touched file:
ruff,pyright,audit_test_purity(theclient.call_llm/_provider_managerswap pattern in some tests — confirm it passes T-rules), and the infra-env rule. - Review Builder 5's diff (it was stopped mid-verify) — confirm each converted qwen/minimax/deepseek/llm-judge test actually passes under synth/replay.
- Memory:
feedback-tests-claude-only.mdupdated; consider a note on the synth mode + observer-in-hook once settled. - Commit only when Mohamed asks (no AI attribution).