providers package

Reviewed against commit 8a2cb42ad on 2026-10-23.

Reference: generated package page orchestrata_ultimate.providers (16 modules including the 8-module adapters subpackage, 2,722 lines of code); key modules provider_manager, router, circuit_breaker, cost_tracker. Its caller is the llm package, which registers one adapter per backend; for the model layer as a whole see the orchestrata overview.

The story

Every volk in the orchestrata pipeline ultimately needs to talk to a large language model, but no single backend can be trusted alone: APIs rate-limit, go down, cost money, and differ in capabilities. orchestrata_ultimate.providers (SPEC-011) exists to absorb all of that operational reality behind one interface, so the rest of the system — the LLM client that volk brains call through, and everything above it — can simply say "run this request" and get an answer from some healthy, affordable, capable backend.

The package's public face is a single class, ProviderManager, re-exported from __init__.py. When a request arrives, the manager consults the ProviderRegistry for the providers and model metadata that have been registered (registration is dynamic — providers can be added and removed at runtime), then hands the candidates to the RequestRouter. The router filters them the way an air-traffic controller clears runways: providers whose CircuitBreaker is open are excluded, providers without rate-limit headroom in the RateLimiter's sliding windows are skipped, capability mismatches are dropped, and budget limits from the CostTracker are respected — only then does a configurable RoutingStrategy pick the winner. The request is executed through that provider's adapter, the outcome (latency, success or failure) is recorded in the HealthMonitor, the spend is booked atomically in Redis by the cost tracker, and failures feed back into the circuit breaker so a flapping backend is temporarily taken out of rotation.

The actual wire-level work lives in the adapters/ subpackage. Anthropic, OpenAI, DeepSeek, Ollama, and OpenRouter adapters all inherit shared HTTP plumbing from BaseAdapter; the Anthropic adapter additionally translates OpenAI-style tool definitions and tool-choice values into Anthropic's format. One adapter breaks the mold: ClaudeCLIAdapter shells out to the Claude Code CLI (claude --print) instead of speaking HTTP, flattening chat messages into a prompt and mapping model aliases to CLI model names — but it still satisfies the same provider protocol, so the router treats it like any other backend.

Collaboration with neighboring packages is deliberately narrow. The routing types and provider protocols (ProviderRequest, ModelCapability, RoutingStrategy, ProviderManagerProtocol) come from core.protocols, tracing decorators come from core (traces), and durable state — circuit-breaker state, cost aggregates, budget counters — is kept in Redis obtained via core.redis_factory, so breaker and budget state survive process restarts. Token estimation borrows memory.token_counter. Upstream, llm/client.py and volk agents consume the manager rather than any adapter directly, which is what lets fallback, budgets, and health policy change without touching a single caller.

Key responsibilities

• Single entry point — ProviderManager presents one execute(request) surface over many heterogeneous LLM backends, hiding adapter details from callers.
• Dynamic provider registry — ProviderRegistry stores providers, configs, and model metadata; supports runtime (de)registration, model-alias resolution, and capability-based lookup.
• Strategy-based routing — RequestRouter filters candidates by availability, rate-limit headroom, capability match, and budget, then ranks survivors by the requested RoutingStrategy (cost / quality / latency / round-robin).
• Fault isolation — CircuitBreaker (closed → open → half-open, Redis-persisted) pulls a failing provider out of rotation and probes it back in after cooldown.
• Rate-limit awareness — RateLimiter tracks RPM/TPM in a sliding window and routes traffic away from providers with low headroom.
• Cost tracking + budget enforcement — CostTracker prices token usage and enforces per-scope budgets atomically via a Redis Lua script; aggregates spend by volk, session and project, recording provider and model per cost.
• Passive health monitoring — HealthMonitor derives P50/P95/P99 latency, error rate, and status from a sliding window of real request outcomes — no synthetic probes.
• Backend adapters — BaseAdapter + HTTP subclasses (Anthropic, OpenAI, DeepSeek, Ollama, OpenRouter) and the subprocess-based ClaudeCLIAdapter, all conforming to ProviderProtocol.

Class diagram

Orchestration core — the manager and its collaborators

ProviderManager composes the registry, router, cost tracker, and health monitor, and owns the per-provider circuit breakers and rate limiters that the router reads while filtering. Generated from the AST and render-verified.

CircuitState is a StrEnum (its three members are shown above); state(), is_available(), headroom_pct(), has_low_headroom(), and provider_names() are Python @property accessors, drawn here as methods for clarity.

Backend adapters — one protocol, many wires

Every backend satisfies the same ProviderProtocol (complete / get_models / supports_capability). The five HTTP backends share BaseAdapter; ClaudeCLIAdapter implements the protocol independently over a subprocess. Generated from the AST and render-verified.

Inheritance across the package

The whole-package inheritance picture, including the exception vocabulary and the CircuitState enum (generated from the AST, render-verified):

Module dependency diagram

Internal module imports within orchestrata_ultimate.providers (generated from the AST, render-verified):

Module inventory

provider_manager.py

This is the façade of the package and the only module callers should import. ProviderManager implements ProviderManagerProtocol and wires every other module together: it consults the ProviderRegistry for available providers, asks the RequestRouter to pick one, checks the chosen provider's CircuitBreaker and RateLimiter, enforces budgets through the CostTracker, and records each outcome in the HealthMonitor. It also defines the package's error vocabulary — ProviderError for backend failures and BudgetExceededError when a request would breach a budget — so callers handle failures without knowing which backend was involved.

router.py

RequestRouter (FR-003) is the decision-maker that turns a ProviderRequest into a concrete provider choice. It builds its candidate set from the registry, then progressively filters: availability (health and circuit-breaker state), rate-limit headroom, capability match (ModelCapability), and budget. Only the survivors are ranked by the configured RoutingStrategy (types imported from core.protocols). Because the router reads breaker and limiter state directly, a provider that just tripped its breaker vanishes from routing on the very next request.

circuit_breaker.py

CircuitBreaker (FR-009) gives each provider the classic closed → open → half-open state machine, modeled by the CircuitState StrEnum, with configurable failure threshold, cooldown, and success threshold for recovery. State is persisted in Redis, so a breaker that opened before a restart stays open afterward, and threading locks keep transitions safe under concurrent requests. The manager reports failures into it; the router reads it when filtering candidates.

cost_tracker.py

CostTracker (FR-011/FR-012) turns raw token usage into money and enforces limits before they are broken. It is Redis-backed, and the critical budget check-and-spend is implemented as a single Redis Lua script so it is atomic — there is no window between "is there budget left?" and "spend it" in which two concurrent requests could both squeeze through. Costs aggregate at volk, session and project scopes (the provider and model are recorded with each cost), with budget_key(scope_key) building the Redis key for each scope.

health_monitor.py

HealthMonitor (FR-007/US-007) is deliberately passive: it never probes a provider, it only listens. Each completed request is recorded as a RequestMetric dataclass in a per-provider sliding window (a collections.deque), and latency, error rate, and availability are derived from that window on demand. The router consumes these derived signals when filtering, which means health follows real traffic rather than synthetic pings.

adapters/

The subpackage where protocol meets wire. All HTTP-based backends subclass BaseAdapter (adapters/base.py), which centralizes shared HTTP logic and parameter normalization such as strip_tools_when_choice_none(). The Anthropic adapter goes further, translating OpenAI-style tool definitions and tool-choice values into Anthropic's native format before dispatch. ClaudeCLIAdapter is the outlier: rather than HTTP it launches the Claude Code CLI as a subprocess (claude --print), locating the binary with _find_claude, flattening the chat history into a single prompt with _build_prompt, and mapping model aliases to CLI model names with _map_model — yet from the router's point of view it is just another provider satisfying the same protocol.

Class map (generated)

Generated from the live source AST by scripts/gen-module-class-diagrams.py — do not hand-edit. Abstract classes have a green border; members listed are those defined on the class itself (inherited members excluded). Solid empty-triangle arrows = in-package inheritance; dashed arrows into dashed boxes = external/stdlib/cross-package bases; filled diamond = composition (constructed in __init__); open diamond = aggregation (annotated/injected reference); diamond edge labels name the owning attribute.