Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate / Specs / 011-provider-manager

011-provider-manager: Provider Manager

The Provider Manager is the unified LLM provider abstraction for Orchestrata. It manages multiple LLM backends (OpenAI, Anthropic, DeepSeek, Ollama, etc.), providing request routing, load balancing, cost tracking, fallback chains, and rate limit management — all governed by PAM++ policies.

At a glance

Item Value
Spec directory specs/011-provider-manager/
Status line in the spec Draft
Requirement IDs defined 60
With a @traces citation in src/ 39 (65.0%)
With a @traces citation in tests/ 1 (1.7%)
Tracked files 9
RTM page 011-provider-manager

Reading this spec

The Provider Manager is the unified LLM provider abstraction for Orchestrata. It manages several LLM backends (OpenAI, Anthropic, DeepSeek, Ollama and others) and provides request routing, load balancing, cost tracking, fallback chains and rate limit management, all governed by PAM++ policies. It exists because Volks depend on specific providers but need resilience when one fails, cost optimization needs per-model pricing awareness, different tasks need different models, and the verification package has only a partial, non-general model registry.

The scope is seven user stories: provider-agnostic execution, cost-aware routing, fallback resilience and rate limit management (priority 1), and model capability matching, budget enforcement and provider health monitoring (priority 2). Requirement IDs use the REQ-011- prefix.

The key decisions are an IProviderManager interface registered in the DI container as ComponentType.PROVIDER_MANAGER; provider adapters that each implement the IProvider protocol; routing based on model capability, cost, latency and availability; fallback chains with configurable retry and circuit breaker patterns; per-request cost tracking with budget enforcement at Volk and project level; and PAM++ governance on model access and budget limits.

In the code, the largest tracers are orchestrata_ultimate/providers/provider_manager, orchestrata_ultimate/core/protocols/providers, orchestrata_ultimate/providers/router, orchestrata_ultimate/providers/cost_tracker and orchestrata_ultimate/providers/rate_limiter. The per-requirement view is on the RTM page.

The spec's status line reads Draft. It tracks 9 files and defines 60 requirement IDs. The full text follows below.

Measured coverage is 39 of 60 requirements cited by at least one @traces in src/ (65.0%), spread over 14 modules; the module citing the most of them is orchestrata_ultimate/core/protocols/providers (17 requirements). In tests/, 1 requirement is cited by a test-side @traces.

Coverage gaps: 21 requirements have no citing symbol. The largest untraced groups are REQ-011-* (21). Untraced means no @traces names the ID; the behaviour may still exist in code that was never annotated. Test-side citations are thinner than code-side ones (1 against 39).

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID Done Requirement Traced by
REQ-011-001 no IProviderManager defined as ABC in orchestrata-core interfaces module 1
REQ-011-002 no IProviderManager registered with ComponentType.PROVIDER_MANAGER in DI container none
REQ-011-003 no IComponent lifecycle implemented (initialize, start, stop, get_health) 7
REQ-011-004 no ProviderRequest data class defined with messages, model, capabilities, parameters 1
REQ-011-005 no ProviderResponse data class defined with content, model_used, provider_used, usage, cost 1
REQ-011-006 no Execute accepts ProviderRequest and returns ProviderResponse 3
REQ-011-007 no Provider selected based on model specification or capability matching 12
REQ-011-008 no Response normalized to common ProviderResponse format 3
REQ-011-009 no ProviderError raised with details when all providers fail 4
REQ-011-010 no Cost recorded for every executed request 4
REQ-011-011 no Routing strategy: cost_optimized implemented 3
REQ-011-012 no Routing strategy: latency_optimized implemented 3
REQ-011-013 no Routing strategy: quality_optimized implemented 3
REQ-011-014 no Routing strategy: round_robin implemented 3
REQ-011-015 no Providers filtered by availability before routing 6
REQ-011-016 no Providers filtered by rate limit headroom before routing 5
REQ-011-017 no Routing decisions logged for auditability none
REQ-011-018 no Cost estimated before execution based on input token count 4
REQ-011-019 no Actual cost recorded after execution based on usage 2
REQ-011-020 no Cumulative cost tracked per volk_id 3
REQ-011-021 no Cumulative cost tracked per session_id 3
REQ-011-022 no Cumulative cost tracked per project_id and portfolio_id 1
REQ-011-023 no Cost queries supported with time range filters 2
REQ-011-024 no Dynamic provider registration and deregistration 9
REQ-011-025 no Provider configuration validated on registration 2
REQ-011-026 no YAML configuration loading from configs/production.yaml none
REQ-011-027 no Environment variable interpolation for secrets none
REQ-011-028 no Model registry with capabilities, context_window, pricing 11
REQ-011-029 no IProvider protocol defined for adapter implementations 1
REQ-011-030 no OpenAI adapter implemented 1
REQ-011-031 no Anthropic adapter implemented 4
REQ-011-032 no DeepSeek adapter implemented 1
REQ-011-033 no Ollama adapter implemented 1
REQ-011-034 no Custom adapter registration supported none
REQ-011-035 no Fallback chains configurable per capability class none
REQ-011-036 no Fallback triggered on timeout, rate limit, server error 1
REQ-011-037 no Fallback NOT triggered on auth error, invalid request, budget exceeded none
REQ-011-038 no Circuit breaker per provider (closed/open/half-open) 7
REQ-011-039 no Circuit opens after configurable consecutive failures (default: 5) 3
REQ-011-040 no Half-open probe after configurable cooldown (default: 60s) 4
REQ-011-041 no Per-provider RPM and TPM tracking 10
REQ-011-042 no Rate limit headers parsed from provider responses 1
REQ-011-043 no Request queuing when rate limit headroom < 10% 2
REQ-011-044 no Alternative provider routing when primary rate-limited 1
REQ-011-045 no Budget limits configurable at Volk, project, portfolio level 8
REQ-011-046 no BudgetExceededError raised when budget exhausted 2
REQ-011-047 no Budget warning events at 80%, 90%, 95% thresholds none
REQ-011-048 no PAM++ governance overrides for emergency escalations none
REQ-011-049 no Model access policies enforced via Prolog governance none
REQ-011-050 no Per-Volk model allowlists and denylists supported none
REQ-011-051 no All provider operations logged to audit trail none
REQ-011-052 no Audit records stored in Neo4j for compliance queries none
REQ-011-053 no Audit queries by time range, volk, provider, cost threshold none
REQ-011-054 no Provider selection < 5ms (P95) none
REQ-011-055 no Cost estimation < 2ms (P95) none
REQ-011-056 no Full request overhead < 20ms (P95) none
REQ-011-057 no Concurrent requests: 200+ simultaneous none
REQ-011-058 no API keys in environment variables, never in code or logs none
REQ-011-059 no API keys excluded from audit records and error messages none
REQ-011-060 no Provider credentials rotatable without restart none

Code that traces this spec

Module Symbols Requirements
orchestrata_ultimate/core/protocols/base 1 1
orchestrata_ultimate/core/protocols/providers 16 17
orchestrata_ultimate/providers/adapters/anthropic_adapter 7 4
orchestrata_ultimate/providers/adapters/claude_cli_adapter 3 3
orchestrata_ultimate/providers/adapters/deepseek_adapter 3 3
orchestrata_ultimate/providers/adapters/ollama_adapter 3 3
orchestrata_ultimate/providers/adapters/openai_adapter 3 3
orchestrata_ultimate/providers/adapters/openrouter_adapter 2 2
orchestrata_ultimate/providers/circuit_breaker 9 3
orchestrata_ultimate/providers/cost_tracker 8 5
orchestrata_ultimate/providers/provider_manager 23 17
orchestrata_ultimate/providers/rate_limiter 11 4
orchestrata_ultimate/providers/registry 6 3
orchestrata_ultimate/providers/router 7 8

Checklists

checklists/constitutional.md

Provider Manager Constitutional Checklist

Security and Governance Principles
Data Integrity and Consistency
Privacy and Data Protection
System Reliability and Resilience
Ethical and Responsible AI
Compliance and Regulatory Requirements
Performance and Resource Management
Interoperability and Standards
Transparency and Explainability
Innovation and Future-Proofing
Human-AI Collaboration

checklists/requirements.md

The requirement list above comes from this file; 0 of 60 checkbox items are ticked.

checklists/testing.md

Provider Manager Testing Checklist

Core Interface Testing
Provider Registration Testing
Request Execution Testing
Routing Strategy Testing
Fallback Chain Testing
Circuit Breaker Testing
Rate Limit Testing
Cost Tracking Testing
Budget Enforcement Testing
Governance Testing
Performance Testing
Security Testing
Integration Testing

Other tracked files

Full spec text

Verbatim from specs/011-provider-manager/spec.md; headings are pushed down two levels.

SPEC-011: Provider Manager

Metadata

Overview

What

The Provider Manager is the unified LLM provider abstraction for Orchestrata. It manages multiple LLM backends (OpenAI, Anthropic, DeepSeek, Ollama, etc.), providing request routing, load balancing, cost tracking, fallback chains, and rate limit management — all governed by PAM++ policies.

Why
How

User Scenarios & Stories

US-001: Provider-Agnostic Execution (P1)

As a Volk I want to send LLM requests without knowing which provider serves them So that I focus on my task rather than provider-specific API details

Acceptance Criteria:

US-002: Cost-Aware Routing (P1)

As the Orchestrator (U:Ox) I want to route requests to the cheapest provider that meets quality requirements So that operational costs are minimized without sacrificing output quality

Acceptance Criteria:

US-003: Fallback Resilience (P1)

As the system I want to automatically retry on alternative providers when one fails So that agent execution is not blocked by transient provider outages

Acceptance Criteria:

US-004: Rate Limit Management (P1)

As the system I want to respect provider rate limits without failing requests So that Orchestrata maintains good standing with API providers

Acceptance Criteria:

US-005: Model Capability Matching (P2)

As the Kernel I want to specify required capabilities (reasoning, coding, vision, tool_use) rather than specific model names So that the system picks the best model for the job

Acceptance Criteria:

US-006: Budget Enforcement (P2)

As the Orchestrator (U:Ox) I want to enforce spending limits per Volk and per project So that runaway agents do not exhaust the budget

Acceptance Criteria:

US-007: Provider Health Monitoring (P2)

As the system I want to track provider health metrics (latency, error rate, availability) So that routing decisions are informed by real-time provider status

Acceptance Criteria:

Functional Requirements

Core Interface

FR-001: IProviderManager Interface (P1)

FR-002: Execute Request (P1)

FR-003: Route Request (P1)

FR-004: Cost Estimation (P1)

Provider Registry

FR-005: Provider Registration (P1)

FR-006: Model Registry (P1)

FR-007: Provider Adapters (P1)

Resilience

FR-008: Fallback Chain (P1)

FR-009: Circuit Breaker (P1)

FR-010: Rate Limit Management (P1)

Cost Management

FR-011: Cost Tracking (P1)

FR-012: Budget Enforcement (P2)

Governance

FR-013: PAM++ Integration (P1)

FR-014: Audit Trail (P1)

FR-015: Token Counting Strategy (P1)

FR-016: Request Cancellation (P2)

FR-017: Streaming Support (Deferred — P3)

Non-Functional Requirements

Performance

NFR-001: Routing Latency

NFR-002: Throughput

Reliability

NFR-003: Availability

NFR-004: Data Integrity

Scalability

NFR-005: Provider Scaling

Security

NFR-006: Credential Management

Technical Design

Architecture
+-------------------------------------------------------------+
|                  Provider Manager System                      |
+-------------------------------------------------------------+
|  +--------------+  +--------------+  +--------------+        |
|  |   Request    |  |    Cost      |  |   Health     |        |
|  |   Router     |  |   Tracker    |  |   Monitor    |        |
|  +--------------+  +--------------+  +--------------+        |
|         |                 |                 |                 |
|         v                 v                 v                 |
|  +-----------------------------------------------------+    |
|  |             Provider Registry                         |    |
|  |  +------+ +----------+ +----------+ +------+         |    |
|  |  |OpenAI| | Anthropic| | DeepSeek | |Ollama|         |    |
|  |  +------+ +----------+ +----------+ +------+         |    |
|  +-----------------------------------------------------+    |
|  +-----------------------------------------------------+    |
|  |         Resilience Layer                              |    |
|  |  Fallback Chain | Circuit Breaker | Rate Limiter      |    |
|  +-----------------------------------------------------+    |
+-------------------------------------------------------------+
Key Interfaces
class IProviderManager(IComponent, ABC):
    """Unified LLM provider abstraction."""

    @abstractmethod
    async def execute(
        self,
        request: ProviderRequest,
    ) -> ProviderResponse:
        """Execute an LLM request. Handles routing, fallback, cost tracking."""

    @abstractmethod
    async def estimate_cost(
        self,
        request: ProviderRequest,
    ) -> CostEstimate:
        """Estimate cost for a request without executing it."""

    @abstractmethod
    async def get_usage(
        self,
        *,
        volk_id: str | None = None,
        session_id: str | None = None,
        project_id: str | None = None,
        date_from: datetime | None = None,
        date_to: datetime | None = None,
    ) -> UsageReport:
        """Query cumulative usage and cost."""

    @abstractmethod
    def register_provider(
        self,
        name: str,
        provider: IProvider,
        config: ProviderConfig,
    ) -> None:
        """Register a provider adapter with configuration."""

    @abstractmethod
    def deregister_provider(self, name: str) -> bool:
        """Remove a provider. Returns True if found and removed."""

    @abstractmethod
    async def get_remaining_budget(
        self,
        *,
        volk_id: str | None = None,
        project_id: str | None = None,
        portfolio_id: str | None = None,
    ) -> BudgetStatus:
        """Query remaining budget for a scope."""

    @abstractmethod
    async def cancel(self, request_id: str) -> bool:
        """Cancel an in-flight request. Returns True if successfully cancelled."""

    @abstractmethod
    def get_provider_health(self) -> dict:
        """Return health status of all registered providers with per-provider metrics."""


class IProvider(Protocol):
    """Protocol for provider adapter implementations."""

    async def complete(
        self,
        messages: list[Message],
        model: str,
        parameters: dict,
    ) -> RawProviderResponse:
        """Send completion request to provider."""

    def get_models(self) -> list[ModelInfo]:
        """Return available models with metadata."""

    def supports_capability(self, capability: ModelCapability) -> bool:
        """Check if provider supports a capability."""


class ModelCapability(Enum):
    REASONING = "reasoning"
    CODING = "coding"
    VISION = "vision"
    TOOL_USE = "tool_use"
    LONG_CONTEXT = "long_context"
    EMBEDDING = "embedding"
    SUMMARIZATION = "summarization"


class RoutingStrategy(Enum):
    COST_OPTIMIZED = "cost_optimized"       # Cheapest eligible provider
    LATENCY_OPTIMIZED = "latency_optimized" # Lowest P95 latency provider
    QUALITY_OPTIMIZED = "quality_optimized"  # Highest quality_rank provider
    ROUND_ROBIN = "round_robin"             # Rotate across eligible providers


class ProviderRequest:
    messages: list[Message]
    model: str | None = None
    capabilities: list[ModelCapability] | None = None
    parameters: dict | None = None
    routing_strategy: RoutingStrategy = RoutingStrategy.COST_OPTIMIZED
    max_tokens: int | None = None
    volk_id: str | None = None
    session_id: str | None = None
    project_id: str | None = None


class ProviderResponse:
    content: str
    model_used: str
    provider_used: str
    usage: TokenUsage
    cost: CostRecord
    latency_ms: float
    metadata: dict


class TokenUsage:
    input_tokens: int
    output_tokens: int
    total_tokens: int


class CostRecord:
    input_cost: float
    output_cost: float
    total_cost: float
    currency: str = "USD"


class CostEstimate:
    estimated_input_tokens: int
    estimated_output_tokens: int
    estimated_cost: CostRecord
    provider: str
    model: str


class BudgetStatus:
    scope: str              # "volk" | "project" | "portfolio"
    scope_id: str
    budget_limit_usd: float
    spent_usd: float
    remaining_usd: float
    usage_percent: float
    warning_threshold: float  # e.g., 0.8
    is_warning: bool
    is_exceeded: bool


class ProviderConfig:
    api_key: str
    base_url: str | None = None
    models: list[str] = []
    rate_limit_rpm: int = 60
    rate_limit_tpm: int = 100_000
    timeout_seconds: float = 30.0
    max_retries: int = 3
    pricing: dict[str, ModelPricing] = {}


class ModelPricing:
    input_per_1m: float          # USD per 1M input tokens
    output_per_1m: float         # USD per 1M output tokens
    capabilities: list[ModelCapability]
    context_window: int
    quality_rank: int = 50       # 1-100, lower = higher quality (used by QUALITY_OPTIMIZED strategy)
Design Notes
Integration Points
Component Integration
orchestrata-core IProviderManager interface, ComponentType.PROVIDER_MANAGER
orchestrata-kernel Provider selection before Volk execution
orchestrata-agents Volk uses provider manager for LLM calls
orchestrata-governance PAM++ model access policies, budget governance
orchestrata-verification Replaces existing ModelRegistry/FallbackChain
orchestrata-session Cost tracking per session
orchestrata-cxp Provider metadata in CXP packet metadata dict
configs/production.yaml Provider configuration source
Kernel Provider Selection

The Kernel selects a provider at the existing governance check point in runtime.py, after Volk definition lookup and before execution begins.

Selection flow:

  1. Volk definition specifies capabilities (e.g., [REASONING, CODING]) or explicit model name
  2. Kernel calls provider_manager.execute(request) — routing is internal to Provider Manager
  3. Provider Manager filters eligible providers by: capability match → availability → rate limit headroom → budget remaining
  4. Applies RoutingStrategy (default: COST_OPTIMIZED) to rank remaining providers
  5. Selected provider + model recorded in CXP packet metadata["provider_selection"]

Configuration per Volk:

# configs/production.yaml
volks:
  "U:Ar":
    preferred_model: "deepseek-reasoner"
    required_capabilities: ["reasoning"]
    budget_limit_usd: 50.0
    routing_strategy: "quality_optimized"
  "U:Vz":
    preferred_model: null  # auto-select
    required_capabilities: ["coding"]
    budget_limit_usd: 100.0
    routing_strategy: "cost_optimized"
  "U:Qa":
    preferred_model: null
    required_capabilities: ["reasoning", "coding"]
    budget_limit_usd: 30.0
    routing_strategy: "cost_optimized"
SPEC-009 Migration Strategy

The existing ModelRegistry and ModelFallbackChain in orchestrata-verification are verification-specific (they expose verify(), analyze(), generate_proof() methods). SPEC-011 generalizes provider management for all Volks.

Migration path: Adapter pattern (no breaking change)

  1. orchestrata-verification keeps its LLMModel protocol and ModelRegistry unchanged
  2. A new VerificationProviderAdapter implements IProvider and wraps LLMModel.verify/analyze/generate_proof as provider completions
  3. ModelFallbackChain is deprecated in favor of SPEC-011's FallbackChain with circuit breaker
  4. Verification package receives IProviderManager via DI instead of constructing its own ModelRegistry
  5. Existing ModelRegistry.register() calls migrate to provider_manager.register_provider()

Timeline:

CXP Integration

Provider metadata uses the existing metadata dict on CXP packets. No schema change to SPEC-004 required.

Convention:

packet.metadata["provider_selection"] = {
    "provider": "deepseek",
    "model": "deepseek-reasoner",
    "routing_strategy": "quality_optimized",
    "fallback_chain": ["deepseek", "anthropic", "openai"],
    "cost_estimate_usd": 0.0042,
    "rate_limit_headroom_pct": 85.0
}
Event Delivery

All provider events are emitted through the Kernel's EventBus (see SPEC-001 EventBus addendum). The Provider Manager does not push to Redis directly — it calls kernel.emit_event().

Emission pattern:

await kernel.emit_event(SystemEvent(
    type="provider.request.completed",
    source="SYS:PROVIDER",
    payload={"request_id": req_id, "provider": provider, "model": model, "cost": cost},
    severity="info",
))

Topic mapping:

Event Topic Severity
Request completed provider.request.completed info
Request failed provider.request.failed error
Fallback triggered provider.fallback.triggered warning
Circuit state changed provider.circuit.state_changed warning
Rate limit approaching provider.ratelimit.approaching warning
Budget warning provider.budget.warning warning
Budget exceeded provider.budget.exceeded critical
Provider registered provider.registered info
Health changed provider.health.changed warning

Reactive subscriptions (registered during initialize()):

# Subscribe to volk termination → cancel in-flight requests
await kernel.event_bus.subscribe(
    "volk.lifecycle.terminated",
    self._on_volk_terminated,
    subscriber_id="SYS:PROVIDER",
)

# Subscribe to session close → migrate cost data to Neo4j
await kernel.event_bus.subscribe(
    "session.closed",
    self._on_session_closed,
    subscriber_id="SYS:PROVIDER",
)

This enables reactive behavior without the Provider Manager needing to know about Volk lifecycle internals or session management — it just subscribes to events.

Redis State Persistence

Circuit breaker state:

orchestrata:provider:cb:{provider_name}:state     — STRING ("closed" | "open" | "half_open")
orchestrata:provider:cb:{provider_name}:failures  — STRING (int, consecutive failure count)
orchestrata:provider:cb:{provider_name}:last_fail — STRING (ISO 8601 timestamp)
orchestrata:provider:cb:{provider_name}:cooldown  — STRING (seconds until half-open probe)

TTL: None (persistent across restarts). Reset on successful half_open → closed transition.

Startup recovery: On Provider Manager initialization, all circuit breaker states are read from Redis. Circuits in open state are immediately transitioned to half_open to trigger a probe on the next request. This prevents circuits from being permanently stuck open after a restart.

Rate limit tracking:

orchestrata:provider:rl:{provider_name}:rpm       — STRING (int, requests this minute)
orchestrata:provider:rl:{provider_name}:tpm       — STRING (int, tokens this minute)
orchestrata:provider:rl:{provider_name}:window    — STRING (ISO 8601, current window start)

TTL: 120s (auto-expires, resets on new window).

Cost aggregation (hot cache):

orchestrata:provider:cost:{volk_id}:{date}        — HASH {total, input, output, requests}
orchestrata:provider:cost:session:{session_id}    — HASH {total, input, output, requests}
orchestrata:provider:cost:project:{project_id}    — HASH {total, input, output, requests}

TTL: None (persistent). Migrated to Neo4j on session close via the same on_session_close hook mechanism as SPEC-010.

Budget atomicity: Budget enforcement uses a Redis Lua script that atomically reads current spend, compares against limit, and increments if within bounds:

-- KEYS[1] = cost hash key, ARGV[1] = estimated_cost, ARGV[2] = budget_limit
local current = tonumber(redis.call('HGET', KEYS[1], 'total') or '0')
local estimate = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
if current + estimate > limit then
    return 0  -- denied
end
redis.call('HINCRBYFLOAT', KEYS[1], 'total', estimate)
return 1  -- allowed

This eliminates TOCTOU race conditions under concurrent load.

Neo4j Audit Schema

Formal audit schema defined in contracts/graph-schema.cql. Nodes:

Relationships:

Prolog Governance Predicates

The Provider Manager extends the PAM++ rule base:

%% Model access governance
allowed_model(VolkId, Model) :-
    registered(VolkId),
    volk_model_allowlist(VolkId, Model).

allowed_model(VolkId, Model) :-
    registered(VolkId),
    \+ volk_model_denylist(VolkId, _),
    model_registered(Model).

%% Budget governance
within_budget(VolkId, EstimatedCost) :-
    volk_budget_limit(VolkId, Limit),
    volk_current_spend(VolkId, Spent),
    Spent + EstimatedCost =< Limit.

%% Emergency escalation override
budget_override(VolkId, _EstimatedCost) :-
    escalation_active(VolkId),
    has_role(VolkId, escalation_authority).

%% Provider access
can_use_provider(VolkId, Provider) :-
    registered(VolkId),
    provider_available(Provider),
    \+ provider_blocked_for(VolkId, Provider).
Rate Limit Header Parsing

Per-provider rate limit header mapping:

Provider RPM Header TPM Header Remaining Header
OpenAI x-ratelimit-limit-requests x-ratelimit-limit-tokens x-ratelimit-remaining-requests
Anthropic anthropic-ratelimit-requests-limit anthropic-ratelimit-tokens-limit anthropic-ratelimit-requests-remaining
DeepSeek x-ratelimit-limit-requests x-ratelimit-limit-tokens x-ratelimit-remaining-requests
Ollama N/A (local) N/A (local) N/A

Configuration:

# configs/production.yaml
runtime:
  providers:
    openai:
      api_key: «redacted»
      base_url: "https://api.openai.com/v1"
      models: ["gpt-4o", "gpt-4o-mini", "o1", "o3-mini"]
      rate_limit_rpm: 500
      rate_limit_tpm: 200000
      timeout_seconds: 60.0
      max_retries: 3
      rate_limit_headers:
        rpm: "x-ratelimit-limit-requests"
        tpm: "x-ratelimit-limit-tokens"
        remaining_rpm: "x-ratelimit-remaining-requests"
      pricing:
        gpt-4o:
          input_per_1m: 2.50
          output_per_1m: 10.00
          capabilities: ["reasoning", "coding", "vision", "tool_use"]
          context_window: 128000
        gpt-4o-mini:
          input_per_1m: 0.15
          output_per_1m: 0.60
          capabilities: ["coding", "tool_use"]
          context_window: 128000
    anthropic:
      api_key: «redacted»
      base_url: "https://api.anthropic.com/v1"
      models: ["claude-opus-4-6", "claude-sonnet-4-6"]
      rate_limit_rpm: 1000
      rate_limit_tpm: 400000
      timeout_seconds: 60.0
      max_retries: 3
      rate_limit_headers:
        rpm: "anthropic-ratelimit-requests-limit"
        tpm: "anthropic-ratelimit-tokens-limit"
        remaining_rpm: "anthropic-ratelimit-requests-remaining"
      pricing:
        claude-opus-4-6:
          input_per_1m: 15.00
          output_per_1m: 75.00
          capabilities: ["reasoning", "coding", "vision", "tool_use", "long_context"]
          context_window: 200000
        claude-sonnet-4-6:
          input_per_1m: 3.00
          output_per_1m: 15.00
          capabilities: ["reasoning", "coding", "vision", "tool_use", "long_context"]
          context_window: 200000
    deepseek:
      api_key: «redacted»
      base_url: "https://api.deepseek.com/v1"
      models: ["deepseek-reasoner", "deepseek-chat"]
      rate_limit_rpm: 300
      rate_limit_tpm: 100000
      timeout_seconds: 120.0
      max_retries: 2
      pricing:
        deepseek-reasoner:
          input_per_1m: 0.55
          output_per_1m: 2.19
          capabilities: ["reasoning"]
          context_window: 65536
        deepseek-chat:
          input_per_1m: 0.27
          output_per_1m: 1.10
          capabilities: ["coding", "tool_use"]
          context_window: 65536
    ollama:
      api_key: null
      base_url: "http://localhost:11434"
      models: ["llama3.1:70b", "codellama:34b"]
      rate_limit_rpm: 0  # unlimited (local)
      rate_limit_tpm: 0  # unlimited (local)
      timeout_seconds: 300.0
      max_retries: 1
      pricing:
        "llama3.1:70b":
          input_per_1m: 0.0
          output_per_1m: 0.0
          capabilities: ["reasoning", "coding"]
          context_window: 131072
Budget Query API

Volks and the orchestrator can query remaining budget:

# On IProviderManager
async def get_remaining_budget(
    self,
    *,
    volk_id: str | None = None,
    project_id: str | None = None,
    portfolio_id: str | None = None,
) -> BudgetStatus:
    """Query remaining budget for a scope."""

class BudgetStatus:
    scope: str              # "volk" | "project" | "portfolio"
    scope_id: str
    budget_limit_usd: float
    spent_usd: float
    remaining_usd: float
    usage_percent: float
    warning_threshold: float  # e.g., 0.8
    is_warning: bool
    is_exceeded: bool

Success Criteria

  1. Interface Complete: IProviderManager implemented with execute, estimate_cost, get_usage, register_provider
  2. Adapters: OpenAI, Anthropic, DeepSeek, Ollama adapters working
  3. Resilience: Fallback chain + circuit breaker handling provider failures transparently
  4. Cost Tracking: Per-request cost recording with aggregation queries
  5. Performance: Routing overhead < 20ms, no measurable impact on LLM latency
  6. Governance: PAM++ enforced on model access and budget limits
  7. Integration: DI-registered, verification package migrated to use IProviderManager

Dependencies

Revision History

Date Version Author Changes
2026-03-09 1.0 Mohamed Initial specification