Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Tests › Reference › llm

Tests reference — tests/llm

tests/llm — 14 test modules, 188 test functions. Source tree: tests/llm at commit 8a2cb42ad. Generated from the tests AST snapshot by scripts/gen-data-pages.py.

Prose pending.

Modules

Module Tests Covers
test_cassette.py 15 4
test_client.py 45 5
test_client_claude_code_routing.py 3 4
test_client_extended.py 17 7
test_client_openai_registration.py 4 3
test_config.py 44 2
test_exceptions.py 4 1
test_haiku_routing.py 9 3
test_init.py 1 1
test_llm_profiles_integration.py 8 4
test_loosened_intent_lru.py 10 0
test_payload_cassette.py 21 4
test_qwen_minimax_e2e.py 7 3

Module details

tests/llm/test_cassette.py

source · 15 tests · 300 lines

Record/replay cassette layer tests (llm/_cassette.py + its client seam).

Covers: llm._cassette, llm.client, llm.exceptions, core.protocols/ (package)

Kind Name Line Description
function test_key_is_deterministic_for_identical_requests 50
function test_key_differs_by_model_and_content 55
function test_key_includes_temperature_only_when_set 62
function test_save_then_load_round_trips_the_response 75
function test_load_missing_cassette_returns_none 86
function test_load_corrupt_cassette_returns_none_not_raises 90
function test_save_never_raises_into_caller_on_unwritable_directory 98
function test_client_replays_cassette_without_calling_provider 122
function test_client_replay_miss_raises_loudly 141
function test_client_record_mode_captures_once_then_replays 153
function test_content_key_is_model_agnostic 195
function test_client_replays_claude_answer_for_a_different_model 208
function test_record_saves_both_exact_and_content_keys 235
function test_synth_mode_returns_viable_text_on_miss_without_provider_call 264
function test_recorded_cassette_wins_over_synth 283

tests/llm/test_client.py

source · 45 tests · 698 lines

Test Standards Compliance (see AGENTS.md § Test Standards):

Covers: llm.client, llm.config, llm.exceptions, providers.provider_manager, core.protocols/ (package)

Kind Name Line Description
function test_parse_dotenv_file_sets_missing_keys 123
function test_parse_dotenv_file_skips_comments_blank_lines 136
function test_parse_dotenv_file_does_not_overwrite_existing 145
function test_parse_dotenv_file_empty_value_ignored 153
function test_load_dotenv_no_crash_when_cwd_has_env 167
function test_load_dotenv_includes_existing_os_environ 173
function test_load_dotenv_handles_missing_files 188
function test_load_dotenv_swallows_parse_errors 202
function test_register_claude_cli_noop_without_binary 223
function test_register_anthropic_noop_without_api_key 228
function test_register_deepseek_noop_without_api_key 239
function test_register_openrouter_noop_without_api_key 244
function test_register_ollama_silently_skips_when_unconfigured 249
function test_register_adapters_does_not_raise 254
function test_get_provider_manager_returns_singleton 264
function test_get_provider_manager_registers_adapters 271
function test_is_transient_true_for_retry_markers 301
function test_is_transient_false_for_non_retry_errors 316
function test_is_transient_with_valueerror 321
function test_no_eligible_provider_config_verdict_pinned_non_retryable 326
function test_no_eligible_provider_rate_limited_stays_retryable 348
function test_cli_timeout_string_is_transient_hence_needs_budget 360
function test_stop_retrying_when_budget_exhausted 368
function test_fast_transients_keep_all_attempts_inside_budget 380
function test_non_transient_stops_immediately_regardless_of_budget 397
function test_retry_configuration_constants 408
function test_execute_with_retry_wraps_non_transient_error 421
function test_execute_with_retry_chains_original_exception 435
function test_call_llm_wraps_errors_when_model_unknown 452
function test_call_llm_unknown_volk_raises_without_default_entry 465
function test_call_llm_injects_graph_context_into_prompt 493
function test_call_reasoner_uses_u_ar_volk_default 507
function test_call_reasoner_accepts_volk_override 520
function test_call_reasoner_resolves_volk_model_when_no_explicit_model 534
function test_call_llm_with_graph_context_nonempty_no_entities 547
function test_call_llm_returns_content_with_real_provider 568
function test_register_anthropic_registers_when_key_set 593
function test_register_deepseek_registers_when_key_set 604
function test_register_deepseek_returns_when_key_missing 612
function test_register_anthropic_returns_when_key_missing 619
function test_register_openrouter_registers_when_key_set 624
function test_register_openrouter_returns_when_key_missing 632
function test_register_ollama_registers_when_url_set 637
function test_call_reasoner_with_volk_resolves_model_via_get_volk_model 654
function test_call_llm_with_graph_context_entities_triggers_validator 677

tests/llm/test_client_claude_code_routing.py

source · 3 tests · 139 lines

Wiring test for the Claude builder track (SPEC-025 Phase 2c): does a

Covers: llm.client, llm.config, providers.provider_manager, core.protocols/ (package)

Kind Name Line Description
function test_claude_code_profile_row_model_is_a_bare_cli_alias 80 Both existing profiles routing J:Junior through claude_code
function test_model_alias_resolves_to_the_claude_cli_provider 98 The registry indexes models by bare model string, not by the YAML
function test_call_llm_routing_resolves_for_claude_code_profile_row 116 End-to-end proof at the config/resolution layer (no live LLM call):

tests/llm/test_client_extended.py

source · 17 tests · 773 lines

Test Standards Compliance (see AGENTS.md § Test Standards):

Covers: llm.client, llm.exceptions, providers.adapters.base, providers.adapters.ollama_adapter, providers.provider_manager, providers.rate_limiter, core.protocols/ (package)

Kind Name Line Description
function test_provider_manager 148 Return a fresh ProviderManager installed as the module singleton.
function test_load_dotenv_catches_parse_error 186
function test_execute_with_retry_retries_transient_errors 202
function test_execute_with_retry_exhausts_all_retries 228
function test_execute_with_retry_rate_limit_backs_off_past_window 261
function test_call_llm_graph_context_passes_on_first_response 324
function test_call_llm_graph_context_passes_on_validation_retry 370
function test_call_llm_graph_context_fails_after_max_retries 430
function test_call_reasoner_propagates_graph_context 497
function test_capture_exchange_writes_jsonl_on_success 543
function test_capture_exchange_swallows_write_failure 605
function test_call_llm_resolves_model_via_volk_when_model_omitted 655
function test_call_reasoner_resolves_model_via_given_volk 673
function test_call_reasoner_resolves_model_via_default_architect_volk 683
function test_call_with_tools_wraps_error_with_explicit_model 700
function test_call_with_tools_resolves_model_via_volk_when_model_omitted 716
function test_call_with_tools_returns_provider_response_on_success 731

tests/llm/test_client_openai_registration.py

source · 4 tests · 84 lines

Wiring tests for A2: OpenAIAdapter registration in llm/client.py.

Covers: llm.client, providers.adapters.openai_adapter, providers.provider_manager

Kind Name Line Description
function test_register_openai_noop_without_api_key 45 With no OPENAI_API_KEY, _register_openai returns without registering.
function test_register_openai_registers_when_key_set 51 With OPENAI_API_KEY present, a real OpenAIAdapter is registered.
function test_register_openai_respects_explicit_env_over_process_environ 62 The explicit env mapping is authoritative, not os.environ directly.
function test_register_adapters_includes_openai_when_key_set 74 End-to-end: _register_adapters() reaches _register_openai via explicit env.

tests/llm/test_config.py

source · 44 tests · 584 lines

Tests for profile-based LLM config — no hardcoded defaults, pure YAML profiles.

Covers: core.base.traces, llm.config

Kind Name Line Description
function test_default_profile_is_minimax_stack 40 The default profile mirrors openrouter-minimax (2026-07-14): the
function test_deepseek_profile_preserves_legacy_default 56 The pre-2026-07-14 default lives on as SPIKE_PROFILE=deepseek.
function test_get_volk_model_none_falls_back_to_default_sentinel 68 SPEC-028 REQ-028-021/028: a None volk (an unresolved/unstamped
function test_get_volk_provider_none_falls_back_to_default_sentinel 77 Mirrors test_get_volk_model_none_falls_back_to_default_sentinel for
function test_get_volk_model_absent_default_entry_raises_clear_error 85 When the active profile has no default sentinel entry at all
function test_all_profiles_load 98
function test_active_profile_is_default 112 Selecting default makes it the active profile. The ambient
function test_switch_to_claude 124
function test_switch_to_kimi 138
function test_switch_to_budget 145
function test_switch_to_openrouter_cheap 152 Cost-tier split (user-selected 2026-07-13): orchestration volks on
function test_switch_to_openrouter_minimax 165 Orchestration-reliability revision (user-selected 2026-07-13 after
function test_switch_to_claude_hybrid 180 Claude builder track: subscription Claude Code (via the claude_code
function test_switch_to_mix 200 mix = Claude for design volks, DeepSeek models via OpenRouter for the
function test_switch_invalid_raises 215
function test_roundtrip_profile_switch 220
function test_unknown_agent_falls_back_to_default_sentinel 231 REQ-028-021: 'a sentinel default key ... applies to any unrecognised
function test_unknown_agent_model_falls_back_to_default_sentinel 239
function test_unknown_agent_raises_when_profile_has_no_default_entry 244 The sentinel fallback is per-profile YAML data, not a code-level
function test_all_agents_have_assignments 262
function test_volk_config_has_known_agents 281
function test_known_agents_excludes_default_sentinel 289 The 'default' sentinel (REQ-028-021/028) is a fallback bucket, not a
function test_documentor_agent_configured 298
function test_orchestrator_agent_configured 303
function test_escalate_to_reasoning_tier 311
function test_escalate_already_at_top_is_noop 322
function test_deepseek_flash_to_pro 333
function test_ollama_escalation_noop 346
function test_provider_models_match_profiles 358
function test_e2e_claude_escalation_cycle 371
function test_e2e_mix_profile_escalation 383 Escalation is family-aware on openrouter: mix's PM (deepseek-v4-flash
function test_escalation_openrouter_top_of_family_is_noop 401 A model absent from OPENROUTER_MODEL_ESCALATIONS (minimax-m3, the
function test_find_profiles_path_raises_when_not_found 416 When no configs/spike-profiles.yaml is found by walking up from the
function test_find_profiles_path_uses_the_fallback_when_the_walk_finds_nothing 441 The install-path fallback is reached, and returned, when it exists.
function test_load_profiles_file_not_found 462
function test_load_profiles_invalid_yaml 472
function test_load_profiles_root_not_a_mapping 484
function test_load_profiles_profile_not_a_mapping 496
function test_load_profiles_agent_cfg_not_a_mapping 508
function test_load_profiles_missing_provider_field 520
function test_load_profiles_missing_model_field 532
function test_load_profiles_valid_file_roundtrip 544 A well-formed profile file loads cleanly (no exception, exact data back).
function test_get_active_profile_falls_back_to_default 562
function test_get_active_profile_raises_without_default 569 If SPIKE_PROFILE names an unknown profile AND 'default' itself has been

tests/llm/test_exceptions.py

source · 4 tests · 46 lines

Tests for llm.exceptions.

Covers: llm.exceptions

Kind Name Line Description
class TestLLMExceptions 29
method TestLLMExceptions.test_llm_client_error_is_exception 30
method TestLLMExceptions.test_llm_client_error_no_message 35
method TestLLMExceptions.test_provider_unavailable_is_llm_client_error 39
method TestLLMExceptions.test_provider_unavailable_inheritance_chain 44

tests/llm/test_haiku_routing.py

source · 9 tests · 195 lines

Tests for Claude Haiku 4.5 wiring (SPEC-028/SPEC-032): provider

Covers: llm.client, llm.config, providers.provider_manager

Kind Name Line Description
function test_register_anthropic_lists_haiku_with_pricing 86
function test_register_anthropic_sibling_pricing_untouched 104
function test_register_claude_cli_lists_haiku 121
function test_concierge_routes_to_haiku_in_claude_profile 139
function test_architect_and_codedirector_unchanged_in_claude_profile 145
function test_other_volks_unchanged_in_claude_profile 151
function test_default_profile_concierge_on_minimax 159
function test_other_profiles_concierge_unchanged 173
function test_roundtrip_profile_switch_back_to_claude_keeps_haiku 189

tests/llm/test_init.py

source · 1 tests · 19 lines

Covers: llm/ (package)

Kind Name Line Description
function test_imports 17

tests/llm/test_llm_profiles_integration.py

source · 8 tests · 210 lines

Integration tests — verify call_llm() routes through correct provider.

Covers: llm.client, llm.config, orchestrator.pipeline_runner, llm/ (package)

Kind Name Line Description
function test_default_profile_routes_through_openrouter 62 Default mirrors openrouter-minimax since 2026-07-14 (native DeepSeek
function test_default_profile_architect_uses_minimax_m3 83 Cassette-replayed (llm_replay, policy 2026-08-04) — see
function test_multi_volk_calls_default_profile 96 Cassette-replayed (llm_replay, policy 2026-08-04) — see
function test_claude_profile_routes_through_anthropic 114
function test_escalation_from_standard_to_reasoning 135
function test_kimi_profile_routes_through_openrouter 160 Cassette-replayed (llm_replay, policy 2026-08-04) — see
function test_mix_profile_per_agent_routing 177 Cassette-replayed (llm_replay, policy 2026-08-04) — see
function test_pipeline_smoke_review_workflow 195 Routed Claude-only (policy 2026-08-04): a bare select_profile("default")

tests/llm/test_loosened_intent_lru.py

source · 10 tests · 288 lines

Behavioral assessment of the T2 loosened-intent demo artifact.

Kind Name Line Description
function test_cache_basic_get_set_delete 93
function test_cache_rejects_unknown_policy 113
function test_cache_lru_evicts_least_recently_used 123
function test_cache_fifo_evicts_oldest_insert 137
function test_cache_lfu_evicts_least_frequently_used 151
function test_cache_expiry_uses_injected_clock 172
function test_cache_purge_expired 195
function test_cache_byte_bound_evicts_to_target 220
function test_cache_stats_reflect_operations 239
function test_cache_thread_safety_under_capacity_bound 264

tests/llm/test_payload_cassette.py

source · 21 tests · 277 lines

Tests for the non-chat half of the cassette layer.

Covers: llm._cassette, semantic_graph.embedders, semantic_graph.models, semantic_graph.rerankers

Kind Name Line Description
class TestPayloadCassetteStore 50
method TestPayloadCassetteStore.test_the_key_is_stable_for_an_identical_payload 51
method TestPayloadCassetteStore.test_the_key_changes_with_the_payload 58
method TestPayloadCassetteStore.test_kind_is_part_of_the_key 63 An embedding request and a rerank request carrying identical fields
method TestPayloadCassetteStore.test_a_saved_value_round_trips 72
method TestPayloadCassetteStore.test_a_missing_cassette_is_a_miss 78
method TestPayloadCassetteStore.test_a_corrupt_cassette_is_a_miss_not_a_crash 81 Matches CassetteStore's never-raise-into-the-caller contract.
method TestPayloadCassetteStore.test_saving_into_an_unwritable_directory_does_not_raise 89
class TestSyntheticEmbeddings 98
method TestSyntheticEmbeddings.test_it_is_deterministic 99
method TestSyntheticEmbeddings.test_different_texts_get_different_vectors 102
method TestSyntheticEmbeddings.test_the_requested_dimension_is_honoured 106
method TestSyntheticEmbeddings.test_the_vector_is_unit_normed 109 Every consumer here compares by cosine.
method TestSyntheticEmbeddings.test_a_batch_preserves_input_order 115
class TestTheFourModes 121
method TestTheFourModes.test_no_cassette_dir_runs_live 122 A real script with no cassette configured still talks to the
method TestTheFourModes.test_a_hit_replays_without_calling_live 140
method TestTheFourModes.test_synth_answers_a_miss_without_calling_live 158
method TestTheFourModes.test_record_calls_live_once_and_persists_it 176
method TestTheFourModes.test_a_miss_in_replay_only_mode_raises_rather_than_calling_out 206 The point of the whole layer: a gap is a gap, not a silent bill.
method TestTheFourModes.test_the_chat_keys_still_work_as_a_fallback 225 A caller that only knows LLM_CASSETTE_DIR is still covered.
class TestTheProviderPathsAreBehindIt 245
method TestTheProviderPathsAreBehindIt.test_the_embedder_produces_vectors_without_a_usable_key 246 The api key is deliberately nonsense: reaching the provider with it
method TestTheProviderPathsAreBehindIt.test_the_reranker_returns_chunks_without_a_usable_key 258
class TestTheSuiteWideBoundary 271
method TestTheSuiteWideBoundary.test_the_embedding_boundary_is_on_for_every_test 272 conftest sets this at import. Without it, "no real provider call in

tests/llm/test_qwen_minimax_e2e.py

source · 7 tests · 331 lines

E2E integration tests — Qwen and MiniMax profiles through OpenRouter.

Covers: llm.client, llm.config, llm/ (package)

Kind Name Line Description
function test_qwen_plus_code_generation 63 qwen/qwen3.7-plus: routing/mechanics only under cassette replay.
function test_qwen_coder_plus_code_generation 100 qwen/qwen3-coder-plus: routing/mechanics only under cassette replay.
function test_qwen_max_reasoning 134 qwen/qwen3.7-max: reasoning about architecture and race conditions.
function test_qwen_escalation_verified 173 Escalate plus->max and verify the CONFIG-level model change.
function test_minimax_m3_code_generation 222 minimax/minimax-m3: generate a working utility function.
function test_minimax_m3_reasoning 253 minimax/minimax-m3: analyze a system design problem.
function test_cross_model_comparison 290 Same prompt routed to three distinct model identities under replay.

Support files

Fixtures, package markers and helpers without test functions.