Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Tests › Reference › providers

Tests reference — tests/providers

tests/providers — 24 test modules, 312 test functions (cumulative over subdirectories). Source tree: tests/providers at commit 8a2cb42ad. Generated from the tests AST snapshot by scripts/gen-data-pages.py.

Prose pending.

Subdirectories

Directory Modules Tests
adapters/ 9 83

Modules

Module Tests Covers
test_adapters.py 16 6
test_anthropic_adapter.py 8 2
test_base_adapter.py 7 2
test_circuit_breaker.py 14 1
test_circuit_breaker_extended.py 16 1
test_cost_tracker.py 17 2
test_health_monitor.py 9 1
test_provider_defect_fixes.py 14 9
test_provider_manager.py 18 5
test_provider_manager_extended.py 36 3
test_provider_protocol_conformance.py 5 9
test_rate_limiter.py 13 1
test_registry.py 8 2
test_request_router.py 23 5
test_router.py 25 6

Module details

tests/providers/test_adapters.py

source · 16 tests · 202 lines

Tests for provider adapters — real implementations.

Covers: providers.adapters.anthropic_adapter, providers.adapters.deepseek_adapter, providers.adapters.ollama_adapter, providers.adapters.openai_adapter, providers.adapters.openrouter_adapter, core.protocols/ (package)

Kind Name Line Description
class TestOllamaAdapter 35
method TestOllamaAdapter.test_get_headers 36
method TestOllamaAdapter.test_get_models 41
method TestOllamaAdapter.test_supports_capability_yes 54
method TestOllamaAdapter.test_supports_capability_no 64
class TestOpenAIAdapter 71
method TestOpenAIAdapter.test_get_headers 72
method TestOpenAIAdapter.test_get_models 79
method TestOpenAIAdapter.test_supports_capability 92
class TestDeepSeekAdapter 104
method TestDeepSeekAdapter.test_get_headers 105
method TestDeepSeekAdapter.test_get_models 111
method TestDeepSeekAdapter.test_supports_capability 124
class TestAnthropicAdapter 136
method TestAnthropicAdapter.test_get_headers 137
method TestAnthropicAdapter.test_get_models 145
method TestAnthropicAdapter.test_supports_capability 158
class TestOpenRouterAdapter 170
method TestOpenRouterAdapter.test_get_headers 171
method TestOpenRouterAdapter.test_get_models 180
method TestOpenRouterAdapter.test_supports_capability 193

tests/providers/test_anthropic_adapter.py

source · 8 tests · 197 lines

Tests for providers.adapters.anthropic_adapter.

Covers: providers.adapters.anthropic_adapter, core.protocols/ (package)

Kind Name Line Description
class TestAnthropicAdapterHeaders 85
method TestAnthropicAdapterHeaders.test_get_headers 86
class TestAnthropicAdapterModels 95
method TestAnthropicAdapterModels.test_get_models_empty 96
method TestAnthropicAdapterModels.test_get_models_with_pricing 102
class TestAnthropicAdapterCapabilities 121
method TestAnthropicAdapterCapabilities.test_supports_capability_when_present 122
method TestAnthropicAdapterCapabilities.test_supports_capability_when_absent 131
method TestAnthropicAdapterCapabilities.test_supports_capability_empty 137
class TestAnthropicAdapterCompleteStructure 143
method TestAnthropicAdapterCompleteStructure.test_complete_builds_payload_without_system 144 No system message -> no system key, and the messages array is
method TestAnthropicAdapterCompleteStructure.test_complete_builds_payload_with_system 173 A system message is pulled out into the top-level system string

tests/providers/test_base_adapter.py

source · 7 tests · 82 lines

Tests for providers.adapters.base.

Covers: providers.adapters.base, core.protocols/ (package)

Kind Name Line Description
class TestBaseAdapter 27
method TestBaseAdapter.test_init_stores_config 28
method TestBaseAdapter.test_get_headers_default_empty 38
method TestBaseAdapter.test_ensure_client_creates_client 43
method TestBaseAdapter.test_ensure_client_reuses_existing 53
method TestBaseAdapter.test_close_closes_client 63
method TestBaseAdapter.test_close_no_client 72
method TestBaseAdapter.test_build_pricing 78

tests/providers/test_circuit_breaker.py

source · 14 tests · 140 lines

Tests for providers/circuit_breaker.py — pure state machine logic.

Covers: providers.circuit_breaker

Kind Name Line Description
class TestCircuitBreakerInitialState 26
method TestCircuitBreakerInitialState.test_initial_state_is_closed 27
method TestCircuitBreakerInitialState.test_initial_counts_zero 32
method TestCircuitBreakerInitialState.test_custom_thresholds 38
class TestCircuitBreakerRecordFailure 47
method TestCircuitBreakerRecordFailure.test_closed_records_failure 48
method TestCircuitBreakerRecordFailure.test_closed_opens_at_threshold 54
method TestCircuitBreakerRecordFailure.test_half_open_failure_goes_open 62
method TestCircuitBreakerRecordFailure.test_open_stays_open_on_failure 69
class TestCircuitBreakerRecordSuccess 77
method TestCircuitBreakerRecordSuccess.test_closed_resets_failures 78
method TestCircuitBreakerRecordSuccess.test_half_open_counts_success 85
method TestCircuitBreakerRecordSuccess.test_half_open_closes_at_success_threshold 92
method TestCircuitBreakerRecordSuccess.test_open_ignores_success 102
class TestCircuitBreakerCooldownTransition 110
method TestCircuitBreakerCooldownTransition.test_open_transitions_to_half_open_after_cooldown 111
method TestCircuitBreakerCooldownTransition.test_open_stays_open_before_cooldown 118
class TestCircuitBreakerReset 128
method TestCircuitBreakerReset.test_reset_to_closed 129

tests/providers/test_circuit_breaker_extended.py

source · 16 tests · 403 lines

Extended tests for providers/circuit_breaker.py — Redis persistence & edge cases.

Covers: providers.circuit_breaker

Kind Name Line Description
class TestCircuitBreakerEdgeCases 44
method TestCircuitBreakerEdgeCases.test_open_without_last_failure_time_stays_open 45 If OPEN but last_failure_time is None, cooldown never elapses.
method TestCircuitBreakerEdgeCases.test_half_open_is_available 53
method TestCircuitBreakerEdgeCases.test_open_before_cooldown_is_not_available 58
method TestCircuitBreakerEdgeCases.test_state_property_mutates_internal_state 66 Accessing state property can transition OPEN -> HALF_OPEN.
class TestCircuitBreakerRedisPersistence 76
method TestCircuitBreakerRedisPersistence.test_set_redis_loads_empty_state 77 set_redis with no prior data leaves default state.
method TestCircuitBreakerRedisPersistence.test_save_state_writes_to_redis 88 record_failure persists state to Redis.
method TestCircuitBreakerRedisPersistence.test_load_state_restores_closed 107 Manually seed Redis, then load state into a fresh breaker.
method TestCircuitBreakerRedisPersistence.test_load_state_restores_half_open 132 Redis has half_open state with some successes.
method TestCircuitBreakerRedisPersistence.test_load_state_open_transitions_to_half_open 158 On restart, OPEN in Redis transitions to HALF_OPEN to allow probing.
method TestCircuitBreakerRedisPersistence.test_record_success_saves_to_redis 188 Success in HALF_OPEN persists new state to Redis.
method TestCircuitBreakerRedisPersistence.test_record_failure_in_half_open_saves_open 220 Failure in HALF_OPEN immediately reopens and persists.
method TestCircuitBreakerRedisPersistence.test_closed_success_resets_failures_and_saves 248 Success in CLOSED resets failure count and persists.
method TestCircuitBreakerRedisPersistence.test_reset_does_not_clear_redis 275 reset() is synchronous and does not touch Redis.
method TestCircuitBreakerRedisPersistence.test_full_lifecycle_with_redis 302 Complete CLOSED -> OPEN -> HALF_OPEN -> CLOSED cycle with Redis.
method TestCircuitBreakerRedisPersistence.test_load_state_with_no_redis_key 350 Fresh breaker + fresh Redis key should stay default.
class TestCircuitBreakerRedisBytesHandling 362
method TestCircuitBreakerRedisBytesHandling.test_load_state_with_bytes_values 363 Redis returns bytes when decode_responses=False; _load_state handles it.

tests/providers/test_cost_tracker.py

source · 17 tests · 214 lines

Test Standards Compliance (see AGENTS.md § Test Standards):

Covers: providers.cost_tracker, core.protocols/ (package)

Kind Name Line Description
function test_id 53
class TestCostTrackerInit 57
method TestCostTrackerInit.test_init 58
class TestCostTrackerCalculateCost 64
method TestCostTrackerCalculateCost.test_calculate_cost 65
method TestCostTrackerCalculateCost.test_calculate_cost_zero 75
class TestCostTrackerBudget 83
method TestCostTrackerBudget.test_set_budget 85
method TestCostTrackerBudget.test_check_budget_no_limit 91
method TestCostTrackerBudget.test_check_budget_no_redis 96
method TestCostTrackerBudget.test_check_budget_with_redis 102
class TestCostTrackerRecordCost 116
method TestCostTrackerRecordCost.test_record_cost_no_redis 118
method TestCostTrackerRecordCost.test_record_cost_with_redis 124
class TestCostTrackerGetUsage 141
method TestCostTrackerGetUsage.test_get_usage_no_redis 143
method TestCostTrackerGetUsage.test_get_usage_with_redis 150
method TestCostTrackerGetUsage.test_get_usage_no_match 164
class TestCostTrackerGetRemainingBudget 171
method TestCostTrackerGetRemainingBudget.test_no_limit 173
method TestCostTrackerGetRemainingBudget.test_with_limit_no_spent 180
method TestCostTrackerGetRemainingBudget.test_with_limit_and_spent 190
method TestCostTrackerGetRemainingBudget.test_exceeded 207

tests/providers/test_health_monitor.py

source · 9 tests · 87 lines

Tests for providers.health_monitor.

Covers: providers.health_monitor

Kind Name Line Description
class TestHealthMonitor 26
method TestHealthMonitor.test_init 27
method TestHealthMonitor.test_record 31
method TestHealthMonitor.test_record_multiple 37
method TestHealthMonitor.test_get_health_unknown 43
method TestHealthMonitor.test_get_health_healthy 49
method TestHealthMonitor.test_get_health_warning 58
method TestHealthMonitor.test_get_health_degraded 67
method TestHealthMonitor.test_get_all_health 76
method TestHealthMonitor.test_percentile 84

tests/providers/test_provider_defect_fixes.py

source · 14 tests · 533 lines

Regression tests for verified defects in orchestrata_ultimate.providers.

Covers: providers.adapters.base, providers.adapters.claude_cli_adapter, providers.adapters.openai_adapter, providers.adapters.openrouter_adapter, providers.circuit_breaker, providers.cost_tracker, providers.provider_manager, providers.rate_limiter, core.protocols/ (package)

Kind Name Line Description
function test_id 48
class TestCostTrackerCanonicalKey 75
method TestCostTrackerCanonicalKey.test_recorded_spend_is_reflected_in_check_budget 76 record_cost must update the SAME key that check_budget reads.
method TestCostTrackerCanonicalKey.test_check_budget_is_non_mutating 111 check_budget must not itself increment spend (it is a read check).
method TestCostTrackerCanonicalKey.test_project_scope_recorded_to_canonical_key 129 project_id spend also lands on its canonical budget key.
class TestClaudeCLITimeoutKill 166
method TestClaudeCLITimeoutKill.test_adapter_complete_kills_subprocess_on_timeout 167 complete() must kill+reap the child when wait_for times out.
class TestAdapterNullContent 244
method TestAdapterNullContent.test_openai_null_content_becomes_empty_string 245 OpenAIAdapter.complete must return '' (str) when content is null.
method TestAdapterNullContent.test_openrouter_null_content_becomes_empty_string 253
class TestCircuitBreakerConcurrency 311
method TestCircuitBreakerConcurrency.test_concurrent_failures_count_is_exact 312 Many parallel record_failure calls must yield a consistent count.
method TestCircuitBreakerConcurrency.test_state_property_is_threadsafe_during_transition 330 Concurrent reads of .state during a cooldown transition stay valid.
class TestRateLimiterConcurrency 358
method TestRateLimiterConcurrency.test_concurrent_record_request_counts_exact 359 Parallel record_request increments must not be lost to races.
class TestProviderClientClosure 392
method TestProviderClientClosure.test_stop_closes_provider_clients 393 ProviderManager.stop() must aclose every adapter's httpx client.
method TestProviderClientClosure.test_deregister_closes_provider_client 415 deregister_provider must close the removed adapter's client.
class TestExecuteInFlightAndForwarding 465
method TestExecuteInFlightAndForwarding.test_execute_forwards_tools_tool_choice_max_tokens 466 execute() must forward request.max_tokens/tools/tool_choice.
method TestExecuteInFlightAndForwarding.test_execute_registers_in_flight_and_cancel_works 504 execute() must register the in-flight task so cancel() can stop it.

tests/providers/test_provider_manager.py

source · 18 tests · 218 lines

Tests for providers.provider_manager.

Covers: providers.adapters.base, providers.circuit_breaker, providers.provider_manager, providers.registry, core.protocols/ (package)

Kind Name Line Description
class TestProviderError 59
method TestProviderError.test_provider_error_message 60
method TestProviderError.test_provider_error_with_attempts 65
class TestBudgetExceededError 70
method TestBudgetExceededError.test_budget_error_message 71
class TestProviderManagerInit 79
method TestProviderManagerInit.test_init_defaults 80
method TestProviderManagerInit.test_metadata 87
class TestProviderManagerRegister 93
method TestProviderManagerRegister.test_register_provider 94
method TestProviderManagerRegister.test_register_provider_breaker_threshold 108
method TestProviderManagerRegister.test_register_provider_limiter_limits 114
class TestProviderManagerHealth 122
method TestProviderManagerHealth.test_get_health_not_running 123
method TestProviderManagerHealth.test_get_health_running_no_providers 127
method TestProviderManagerHealth.test_get_health_running_with_available_provider 132
method TestProviderManagerHealth.test_get_health_running_with_unavailable_provider 139
class TestProviderManagerLifecycle 149
method TestProviderManagerLifecycle.test_start 150
method TestProviderManagerLifecycle.test_stop 154
method TestProviderManagerLifecycle.test_stop_cancels_in_flight 159
class TestProviderManagerRoutingHonesty 170
method TestProviderManagerRoutingHonesty.test_execute_names_circuit_breaker_rejection 171
method TestProviderManagerRoutingHonesty.test_execute_names_rate_limit_rejection 190
method TestProviderManagerRoutingHonesty.test_execute_names_unknown_model 208

tests/providers/test_provider_manager_extended.py

source · 36 tests · 636 lines

Extended tests for providers.provider_manager.

Covers: providers.adapters.base, providers.provider_manager, core.protocols/ (package)

Kind Name Line Description
function test_estimate_cost_uses_real_tokenizer_not_char_heuristic 615 FR-015: input tokens come from the shared TiktokenCounter, not the old
function test_token_counter_for_caches_and_handles_non_openai 630 Counters are memoised per model; a non-OpenAI model still gets a working
class TestDeregisterProvider 138
method TestDeregisterProvider.test_deregister_success 139
method TestDeregisterProvider.test_deregister_not_found 154
method TestDeregisterProvider.test_deregister_updates_router 158
class TestGetProviderHealth 177
method TestGetProviderHealth.test_empty_registry 178
method TestGetProviderHealth.test_with_registered_providers 182
method TestGetProviderHealth.test_after_health_record 195
class TestSetBudget 213
method TestSetBudget.test_set_budget_delegates 214
class TestGetRemainingBudget 227
method TestGetRemainingBudget.test_empty_scope_returns_default 228
method TestGetRemainingBudget.test_with_scope_no_redis 237
class TestInitialize 252
method TestInitialize.test_initialize_direct 253
method TestInitialize.test_initialize_sets_cost_tracker_redis 266
method TestInitialize.test_initialize_sets_breaker_redis 271
method TestInitialize.test_initialize_sets_limiter_redis 279
class TestStop 288
method TestStop.test_stop_closes_redis 289
method TestStop.test_stop_idempotent 296
class TestCancel 307
method TestCancel.test_cancel_existing_task 308
method TestCancel.test_cancel_done_task 317
method TestCancel.test_cancel_missing_task 327
class TestExecuteRouting 337
method TestExecuteRouting.test_execute_no_route_raises 338
method TestExecuteRouting.test_execute_success 343
method TestExecuteRouting.test_execute_no_volk_id_skips_budget 363
method TestExecuteRouting.test_execute_provider_fails_no_fallback 377
method TestExecuteRouting.test_execute_fallback_to_second_provider 395
method TestExecuteRouting.test_execute_budget_exceeded 418
method TestExecuteRouting.test_execute_records_health_on_failure 445
method TestExecuteRouting.test_execute_records_health_on_success 461
method TestExecuteRouting.test_execute_breaker_records_failure 474
method TestExecuteRouting.test_execute_breaker_records_success 488
class TestEstimateCost 506
method TestEstimateCost.test_estimate_cost_no_route 507
method TestEstimateCost.test_estimate_cost_with_pricing 514
class TestGetUsage 544
method TestGetUsage.test_get_usage_no_redis 545
method TestGetUsage.test_get_usage_with_redis 550
class TestEstimateCostForPricing 579
method TestEstimateCostForPricing.test_calculation 580
method TestEstimateCostForPricing.test_default_max_tokens 602

tests/providers/test_provider_protocol_conformance.py

source · 5 tests · 205 lines

Provider-protocol reconciliation wiring tests.

Covers: llm.client, providers.adapters.anthropic_adapter, providers.adapters.claude_cli_adapter, providers.adapters.deepseek_adapter, providers.adapters.ollama_adapter, providers.adapters.openai_adapter, providers.adapters.openrouter_adapter, providers.provider_manager, core.protocols/ (package)

Kind Name Line Description
function test_http_adapter_satisfies_provider_protocol_and_registers 97 Each HTTP-based adapter satisfies ProviderProtocol and registers via
function test_claude_cli_adapter_satisfies_provider_protocol_and_registers 124 ClaudeCLIAdapter (Part 3, additive) now structurally satisfies
function test_claude_cli_adapter_defaults_to_empty_config_when_none_given 149 Backward compatibility: existing zero-arg/timeout_seconds-only callers
function test_all_six_adapters_structurally_satisfy_provider_protocol 162 All 6 concrete adapter classes (Claude CLI, Anthropic, DeepSeek,
function test_register_adapters_via_real_client_path_is_protocol_compliant 189 llm.client._register_adapters() is the actual production wiring

tests/providers/test_rate_limiter.py

source · 13 tests · 97 lines

Tests for providers.rate_limiter — pure logic, no Redis required.

Covers: providers.rate_limiter

Kind Name Line Description
function test_rate_limiter_defaults 13
function test_headroom_full_at_start 20
function test_is_available_at_start 24
function test_has_low_headroom_at_start 28
function test_record_request_increments_count 33
function test_headroom_decreases_after_requests 39
function test_headroom_zero_when_exhausted 45
function test_low_headroom_triggers 52
function test_window_reset 59
function test_get_status 69
function test_unlimited_rpm 81
function test_unlimited_tpm 87
function test_set_redis 95

tests/providers/test_registry.py

source · 8 tests · 105 lines

Tests for providers/registry.py.

Covers: providers.registry, core.protocols/ (package)

Kind Name Line Description
class TestProviderRegistry 41
method TestProviderRegistry.test_register_and_get 42
method TestProviderRegistry.test_deregister_existing 50
method TestProviderRegistry.test_deregister_missing 56
method TestProviderRegistry.test_resolve_model 60
method TestProviderRegistry.test_get_pricing 66
method TestProviderRegistry.test_find_by_capability 75
method TestProviderRegistry.test_provider_names 93
method TestProviderRegistry.test_model_count 99

tests/providers/test_request_router.py

source · 23 tests · 420 lines

Tests for providers/router.py — pure routing logic.

Covers: providers.circuit_breaker, providers.rate_limiter, providers.registry, providers.router, core.protocols/ (package)

Kind Name Line Description
class TestRouteExplicitModel 69
method TestRouteExplicitModel.test_route_explicit_model 70
method TestRouteExplicitModel.test_route_unknown_model_returns_none 82
class TestRouteCapabilityMatch 94
method TestRouteCapabilityMatch.test_route_by_capability 95
method TestRouteCapabilityMatch.test_route_no_matching_capability 108
class TestRouteAvailabilityFiltering 119
method TestRouteAvailabilityFiltering.test_route_skips_open_circuit 120
method TestRouteAvailabilityFiltering.test_route_skips_rate_limited 140
method TestRouteAvailabilityFiltering.test_route_all_unavailable_returns_none 159
class TestRouteStrategyRanking 174
method TestRouteStrategyRanking.test_cost_optimized_picks_cheapest 175
method TestRouteStrategyRanking.test_quality_optimized_picks_best 192
method TestRouteStrategyRanking.test_latency_optimized_with_health_data 209
method TestRouteStrategyRanking.test_latency_optimized_without_health_returns_unsorted 227
method TestRouteStrategyRanking.test_round_robin_rotates 239
class TestFallbackChain 260
method TestFallbackChain.test_fallback_chain_excludes_tried 261
method TestFallbackChain.test_fallback_chain_skips_unavailable 278
method TestFallbackChain.test_fallback_chain_empty_when_all_excluded 293
class TestNoCandidates 306
method TestNoCandidates.test_empty_registry_returns_none 307
class TestRejectionReasons 313
method TestRejectionReasons.test_rejection_reason_unknown_model 314
method TestRejectionReasons.test_rejection_reason_circuit_breaker_open 326
method TestRejectionReasons.test_rejection_reason_rpm_exhausted 341
method TestRejectionReasons.test_rejection_reason_tpm_exhausted 355
method TestRejectionReasons.test_rejection_reason_multiple_candidates 369
class TestRateLimitBackoff 393
method TestRateLimitBackoff.test_backoff_zero_when_no_rate_limited_candidates 394
method TestRateLimitBackoff.test_backoff_returns_seconds_until_reset 405

tests/providers/test_router.py

source · 25 tests · 231 lines

Tests for providers.router.

Covers: providers.adapters.base, providers.circuit_breaker, providers.rate_limiter, providers.registry, providers.router, core.protocols/ (package)

Kind Name Line Description
class TestRequestRouterRoute 79
method TestRequestRouterRoute.test_route_by_model 80
method TestRequestRouterRoute.test_route_by_capability 85
method TestRequestRouterRoute.test_route_no_model_no_capabilities 90
method TestRequestRouterRoute.test_route_unknown_model 96
method TestRequestRouterRoute.test_route_unmatched_capabilities 101
method TestRequestRouterRoute.test_route_circuit_breaker_open 106
method TestRequestRouterRoute.test_route_rate_limiter_exhausted 115
method TestRequestRouterRoute.test_route_cost_optimized 123
method TestRequestRouterRoute.test_route_quality_optimized 129
method TestRequestRouterRoute.test_route_latency_optimized_no_health_data 135
method TestRequestRouterRoute.test_route_latency_optimized_with_health_data 140
method TestRequestRouterRoute.test_route_round_robin 146
class TestRequestRouterFallbackChain 155
method TestRequestRouterFallbackChain.test_fallback_chain_basic 156
method TestRequestRouterFallbackChain.test_fallback_chain_excludes_tried 161
method TestRequestRouterFallbackChain.test_fallback_chain_skips_open_breaker 166
class TestRequestRouterGetCandidates 177
method TestRequestRouterGetCandidates.test_get_candidates_explicit_model 178
method TestRequestRouterGetCandidates.test_get_candidates_capabilities 184
method TestRequestRouterGetCandidates.test_get_candidates_all 189
class TestRequestRouterRank 195
method TestRequestRouterRank.test_rank_cost_optimized 196
method TestRequestRouterRank.test_rank_quality_optimized 201
method TestRequestRouterRank.test_rank_latency_optimized_no_health 206
method TestRequestRouterRank.test_rank_latency_optimized_with_health 211
method TestRequestRouterRank.test_rank_round_robin 217
method TestRequestRouterRank.test_rank_empty 223
method TestRequestRouterRank.test_rank_default_returns_unsorted 227