Orchestrata Ultimate / Specs / 053-openrouter-provider

053-openrouter-provider: OpenRouter Provider

OpenRouter integration enabling access to all LLM providers (Anthropic, OpenAI, Google, local) with intelligent routing and cost optimization.

At a glance

Reading this spec

This spec defines an OpenRouter provider that gives Orchestrata one abstraction over many language-model vendors, with routing that depends on what kind of agent is asking, automatic fallback when a model fails, and cost tracking with budget limits. The models it names are Anthropic (claude-3-5-sonnet, claude-3-opus, claude-3-haiku), OpenAI (gpt-4o, gpt-4o-mini, gpt-3.5-turbo), Google (gemini-1.5-pro, gemini-1.5-flash) and local Ollama models (llama2, codellama, mistral).

Its three user stories are model selection by agent type, automatic fallback and cost tracking. The functional requirements (FR-001 onward) cover multi-provider support, model routing by agent type, automatic fallback, cost tracking and budget enforcement. Data models describe the provider configuration and the cost record. All 98 requirement IDs are numbered REQ-053-NNN.

Several design decisions are stated precisely. Requests route by agent type (coding, review, research and orchestration), with configurable default models. Each agent type has a default fallback chain: coding goes claude-3-5-sonnet, claude-3-opus, gpt-4o, claude-3-haiku; review goes gpt-4o, claude-3-5-sonnet, gpt-4o-mini; research goes claude-3-opus, claude-3-5-sonnet, gemini-1.5-pro; orchestration goes claude-3-5-sonnet, claude-3-haiku. Retries use a maximum of 3 attempts, a 1.0 second base delay, a 60.0 second cap, an exponential base of 2 and jitter. A malformed response is logged and the fallback model is tried. Model lists are cached for 1 hour, pricing for 24 hours and routing decisions for 1 hour. Cost per 1K tokens is tracked, budget enforcement blocks requests at the configured limits, prompts are not logged, and usage data is anonymized. Cost records are retained for 2 years and usage logs for 1 year.

No code module cites this spec. The requirements traceability matrix lists every ID. The spec's own status line reads Complete and it tracks 2 files. No src/ symbol cites any of the 98 requirements, so measured trace coverage is 0.0%, and in tests/, 0 requirements are cited by a test-side @traces. All 98 requirements have no citing symbol; untraced means no @traces names the ID, and the behaviour may still exist in code that was never annotated.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

No src/ symbol cites a requirement of this spec.

Other tracked files

• specs/053-openrouter-provider/compliance-audit.md

Full spec text

Verbatim from specs/053-openrouter-provider/spec.md; headings are pushed down two levels.

SPEC-053: OpenRouter Provider

Metadata

• Spec ID: 053-openrouter-provider
• Title: OpenRouter Provider - Multi-Provider LLM Abstraction
• Author: Mohamed
• Status: Complete
• Created: 2026-04-08
• Last Updated: 2026-04-09
• Priority: P0
• Version: 1.1.0
• Extends: 001-ultimate-architecture

Overview

OpenRouter integration enabling access to all LLM providers (Anthropic, OpenAI, Google, local) with intelligent routing and cost optimization.

Supported Providers

• Anthropic: claude-3-5-sonnet, claude-3-opus, claude-3-haiku
• OpenAI: gpt-4o, gpt-4o-mini, gpt-3.5-turbo
• Google: gemini-1.5-pro, gemini-1.5-flash
• Local: Ollama models (llama2, codellama, mistral)

REQ-053-001: The provider SHALL give access to the Anthropic (claude-3-5-sonnet, claude-3-opus, claude-3-haiku), OpenAI (gpt-4o, gpt-4o-mini, gpt-3.5-turbo), Google (gemini-1.5-pro, gemini-1.5-flash) and local Ollama (llama2, codellama, mistral) models through one abstraction.

Key Features

• Model routing per agent type
• Automatic fallback
• Cost tracking
• Performance optimization

User Scenarios

US-001: Model Selection by Agent Type

As a developer, I want different models for different agent types so I can optimize for quality and cost.

Acceptance Criteria:

• Each agent type has a default model
• Models can be changed per-agent or globally
• Model changes take effect immediately
• Previous requests complete with old model

REQ-053-002: Each agent type SHALL have a default model.

REQ-053-003: Models SHALL be changeable per-agent or globally.

REQ-053-004: A model change SHALL take effect immediately.

REQ-053-005: Requests already in flight when a model changes SHALL complete with the old model.

US-002: Automatic Fallback

As a system, I want automatic fallback when a model fails so I can maintain system availability.

Acceptance Criteria:

• Fallback chain defined per agent type
• All fallbacks tried before returning error
• Fallback selection logged
• User notified of fallback

REQ-053-006: A fallback chain SHALL be defined per agent type.

REQ-053-007: All fallbacks SHALL be tried before an error is returned.

REQ-053-008: Fallback selection SHALL be logged.

REQ-053-009: The user SHALL be notified of a fallback.

US-003: Cost Tracking

As an admin, I want to track costs per model so I can optimize spending.

Acceptance Criteria:

• Cost per 1K tokens tracked
• Daily/weekly/monthly reports available
• Alerts on budget threshold
• Per-user and per-project tracking

REQ-053-010: Cost per 1K tokens SHALL be tracked.

REQ-053-011: Daily, weekly and monthly cost reports SHALL be available.

REQ-053-012: An alert SHALL be raised on reaching the budget threshold.

REQ-053-013: Cost SHALL be tracked per user and per project.

Terminology

Agent Type

Classification of AI agent based on primary function: coding, review, research, or orchestration.

Fallback Chain

Ordered list of alternative models tried sequentially when primary model fails.

Cost Record

Data structure capturing model usage cost including token counts, timestamps, and attribution.

Provider Priority

Numerical ranking determining model selection order when multiple providers are available.

Budget Threshold

Percentage of spending limit (typically 80%) that triggers user notification alerts.

Data Models

ProviderConfig

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["provider", "api_key", "models"],
  "properties": {
    "provider": {
      "type": "string",
      "enum": ["anthropic", "openai", "google", "ollama"]
    },
    "api_key": {
      "type": "string",
      "description": "Encrypted API key"
    },
    "models": {
      "type": "array",
      "items": {"type": "string"}
    },
    "priority": {
      "type": "integer",
      "description": "Selection priority (lower = higher)"
    }
  }
}

REQ-053-014: A ProviderConfig SHALL require provider (one of anthropic, openai, google, ollama), api_key (an encrypted API key) and models (a list of model names), and SHALL carry an integer priority giving selection priority in which a lower value ranks higher.

CostRecord

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["timestamp", "model", "tokens", "cost"],
  "properties": {
    "timestamp": {
      "type": "string",
      "format": "date-time"
    },
    "model": {
      "type": "string"
    },
    "input_tokens": {
      "type": "integer"
    },
    "output_tokens": {
      "type": "integer"
    },
    "cost": {
      "type": "number"
    },
    "user_id": {
      "type": "string"
    },
    "project_id": {
      "type": "string"
    }
  }
}

REQ-053-015: A CostRecord SHALL require timestamp (date-time), model, tokens and cost, and SHALL carry input_tokens, output_tokens, cost, user_id and project_id.

Fallback Model Priority

Default Chains

FALLBACK_CHAINS = {
    "coding": [
        "claude-3-5-sonnet",
        "claude-3-opus", 
        "gpt-4o",
        "claude-3-haiku"
    ],
    "review": [
        "gpt-4o",
        "claude-3-5-sonnet",
        "gpt-4o-mini"
    ],
    "research": [
        "claude-3-opus",
        "claude-3-5-sonnet",
        "gemini-1.5-pro"
    ],
    "orchestration": [
        "claude-3-5-sonnet",
        "claude-3-haiku"
    ]
}

REQ-053-016: The default fallback chains SHALL be coding = [claude-3-5-sonnet, claude-3-opus, gpt-4o, claude-3-haiku], review = [gpt-4o, claude-3-5-sonnet, gpt-4o-mini], research = [claude-3-opus, claude-3-5-sonnet, gemini-1.5-pro], and orchestration = [claude-3-5-sonnet, claude-3-haiku].

Fallback Selection Logic

async def call_with_fallback(agent_type: str, prompt: str) -> str:
    """Call model with automatic fallback."""
    chain = FALLBACK_CHAINS.get(agent_type, FALLBACK_CHAINS["coding"])
    errors = []
    
    for model in chain:
        try:
            result = await call_model(model, prompt)
            await log_fallback(model, success=True)
            return result
        except ModelError as e:
            errors.append({"model": model, "error": str(e)})
            await log_fallback(model, success=False, error=str(e))
            continue
    
    # All models failed - return partial response with error info
    return await call_fallback_exhausted_response(
        agent_type=agent_type,
        errors=errors,
        original_prompt=prompt
    )

async def call_fallback_exhausted_response(
    agent_type: str,
    errors: list,
    original_prompt: str
) -> dict:
    """Handle case when all fallback models fail."""
    
    # Log critical error
    await log_critical({
        "event": "all_models_failed",
        "agent_type": agent_type,
        "errors": errors,
        "prompt_length": len(original_prompt)
    })
    
    # Notify admin
    await notify_admin({
        "subject": "All LLM models failed",
        "agent_type": agent_type,
        "errors": errors
    })
    
    # Return user-friendly error
    return {
        "status": "error",
        "error": "all_models_failed",
        "message": "All available models are currently unavailable. Please try again later.",
        "agent_type": agent_type,
        "retry_after": 60  # seconds
    }

REQ-053-017: call_with_fallback SHALL try the models of the agent type's chain in order, SHALL use the coding chain when the agent type has no chain of its own, SHALL log every attempt with its success or failure and error, and SHALL return the first successful result.

REQ-053-018: When every model in the chain has failed, the system SHALL log a critical all_models_failed event carrying the agent type, the per-model errors and the prompt length, SHALL notify the admin, and SHALL return an error response carrying status error, error all_models_failed, a user-facing message, the agent type and retry_after of 60 seconds.

Request Retry Logic

Retry Configuration

RETRY_CONFIG = {
    "max_retries": 3,
    "base_delay": 1.0,
    "max_delay": 60.0,
    "exponential_base": 2,
    "jitter": True,
    "retryable_errors": [
        "rate_limit",
        "timeout", 
        "service_unavailable",
        "bad_gateway"
    ]
}

REQ-053-019: Retry SHALL be configured with max_retries 3, base_delay 1.0, max_delay 60.0, exponential_base 2, jitter enabled, and the retryable error set {rate_limit, timeout, service_unavailable, bad_gateway}.

Retry Implementation

async def call_with_retry(
    model: str, 
    prompt: str,
    config: dict = RETRY_CONFIG
) -> str:
    """Call API with exponential backoff retry."""
    
    last_error = None
    
    for attempt in range(config["max_retries"]):
        try:
            return await call_model(model, prompt)
        except ModelError as e:
            if e.code not in config["retryable_errors"]:
                raise
            
            last_error = e
            delay = min(
                config["base_delay"] * (config["exponential_base"] ** attempt),
                config["max_delay"]
            )
            
            if config["jitter"]:
                delay *= random.uniform(0.5, 1.5)
            
            await asyncio.sleep(delay)
    
    raise MaxRetriesExceededError(last_error)

REQ-053-020: An error whose code is not in retryable_errors SHALL be raised to the caller immediately, without retry.

REQ-053-021: The delay before retry attempt n SHALL be min(base_delay * exponential_base ** n, max_delay), multiplied by a uniform random factor in [0.5, 1.5) when jitter is enabled.

REQ-053-022: When max_retries attempts have all failed, the system SHALL raise MaxRetriesExceededError carrying the last error.

Cost Tracking

Cost Calculation

MODEL_PRICING = {
    "claude-3-5-sonnet": {"input": 0.003, "output": 0.015},
    "claude-3-opus": {"input": 0.015, "output": 0.075},
    "claude-3-haiku": {"input": 0.00025, "output": 0.00125},
    "gpt-4o": {"input": 0.0025, "output": 0.01},
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
}

def calculate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    """Calculate cost in USD."""
    pricing = MODEL_PRICING.get(model, {"input": 0, "output": 0})
    return (input_tokens / 1000 * pricing["input"]) + \
           (output_tokens / 1000 * pricing["output"])

REQ-053-023: MODEL_PRICING SHALL hold the per-1K-token USD input/output prices claude-3-5-sonnet 0.003/0.015, claude-3-opus 0.015/0.075, claude-3-haiku 0.00025/0.00125, gpt-4o 0.0025/0.01, gpt-4o-mini 0.00015/0.0006.

REQ-053-024: Cost in USD SHALL be computed as input_tokens / 1000 * input_price + output_tokens / 1000 * output_price, using zero pricing for a model absent from MODEL_PRICING.

Budget Alerts

async def check_budget(user_id: str, amount: float):
    """Check if spending exceeds budget."""
    daily_limit = await get_user_budget(user_id)
    current_spend = await get_daily_spend(user_id)
    
    threshold = daily_limit * 0.8  # 80% threshold
    
    if current_spend + amount > threshold:
        await notify_user(user_id, "Budget threshold reached")
    
    if current_spend + amount > daily_limit:
        raise BudgetExceededError("Daily budget exceeded")

REQ-053-025: When a user's current spend plus the request amount exceeds 80% of that user's daily limit, the system SHALL notify the user that the budget threshold has been reached.

REQ-053-026: When a user's current spend plus the request amount exceeds that user's daily limit, the system SHALL raise BudgetExceededError.

Functional Requirements

FR-001: Multi-Provider Support

REQ-053-027: The system MUST support multiple LLM providers including Anthropic, OpenAI, Google, and local Ollama models through a unified abstraction layer.

FR-002: Model Routing by Agent Type

REQ-053-028: The system MUST route requests to appropriate models based on agent type (coding, review, research, orchestration) with configurable default models.

FR-003: Automatic Fallback

REQ-053-029: The system MUST automatically fallback to alternative models when the primary model fails, following predefined fallback chains per agent type.

FR-004: Cost Tracking

REQ-053-030: The system MUST track costs per model, user, and project, including input/output token counts and calculated USD amounts.

FR-005: Budget Enforcement

REQ-053-031: The system MUST enforce daily/weekly/monthly budget limits per user and project, blocking requests that exceed thresholds.

FR-006: Budget Alerts

REQ-053-032: The system MUST send alerts when spending reaches 80% of budget and notify users when limits are exceeded.

FR-007: Retry Logic

REQ-053-033: The system MUST implement exponential backoff retry logic for transient failures including rate limits, timeouts, and service unavailability.

FR-008: Request Validation

REQ-053-034: The system MUST validate all incoming requests including model name verification, context length limits, and budget checks before processing.

FR-009: API Key Management

REQ-053-035: The system MUST encrypt and securely store API keys with quarterly rotation support and usage anomaly monitoring.

FR-010: Audit Logging

REQ-053-036: The system MUST log all model calls with timestamps, user IDs, model names, token counts, and costs for compliance and debugging.

FR-011: Performance Caching

REQ-053-037: The system MUST cache model lists (1 hour), pricing data (24 hours), and routing decisions (1 hour) to minimize latency.

FR-012: Response Time Targets

REQ-053-038: The system MUST deliver cached model responses in <100ms, uncached in <2s, and fallback switches in <500ms.

Error Conditions

REQ-053-039: On ERR-001 (rate limit exceeded) the system SHALL retry with exponential backoff per RETRY_CONFIG.

REQ-053-040: On ERR-002 (model unavailable) the system SHALL fall back to the next model in the chain per FALLBACK_CHAINS.

REQ-053-041: On ERR-003 (invalid API key) the system SHALL raise an error and prompt the user to reconfigure the provider.

REQ-053-042: On ERR-004 (request timeout) the system SHALL retry with exponential backoff for at most 3 attempts.

REQ-053-043: On ERR-005 (context length exceeded) the system SHALL truncate the prompt or split it into multiple requests.

REQ-053-044: On ERR-006 (budget exceeded) the system SHALL block the request, notify the user, and return a budget error.

REQ-053-045: On ERR-007 (service unavailable) the system SHALL fall back to the next provider in priority order.

REQ-053-046: On ERR-008 (malformed response) the system SHALL log the error and attempt the fallback model.

REQ-053-047: On ERR-009 (authentication failure) the system SHALL revoke the token and prompt re-authentication.

REQ-053-048: On ERR-010 (network connectivity error) the system SHALL retry with backoff and log the connectivity issue.

REQ-053-049: On ERR-011 (provider throttling) the system SHALL apply the provider-specific delay and queue the request.

REQ-053-050: On ERR-012 (all fallback models exhausted) the system SHALL return a partial response carrying the error info and SHALL notify the admin.

Visual Architecture

Class Diagram

Diagram source shown as text: Mermaid could not parse it (Error: Parse error on line 9:).

classDiagram-direction LR
    class OpenRouterProvider {
        +__init__(api_key: str | None = None)
        +async chat_completion(model: str, messages: list[dict[str, str]], **kwargs): dict[str, Any]
        +get_available_models(): list[str]
        +get_model_cost(model: str): float
    }

REQ-053-051: OpenRouterProvider SHALL expose __init__(api_key: str | None = None), async chat_completion(model: str, messages: list[dict[str, str]], **kwargs) -> dict[str, Any], get_available_models() -> list[str] and get_model_cost(model: str) -> float.

Model API

GET  /api/v1/models                    # List available models
GET  /api/v1/models/{id}              # Get model details
POST /api/v1/completions              # Generate completion
GET  /api/v1/costs                    # Get cost reports

REQ-053-052: The system SHALL expose GET /api/v1/models (list available models), GET /api/v1/models/{id} (get model details), POST /api/v1/completions (generate completion) and GET /api/v1/costs (get cost reports).

Request Format

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {"role": "user", "content": "Hello"}
  ],
  "temperature": 0.7,
  "max_tokens": 1000
}

REQ-053-053: A completion request SHALL carry model and messages (each message a role/content pair), and SHALL accept temperature and max_tokens.

Response Format

{
  "id": "gen-123",
  "model": "claude-3-5-sonnet",
  "choices": [
    {"message": {"role": "assistant", "content": "Hello!"}}
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 8,
    "total_tokens": 18
  }
}

REQ-053-054: A completion response SHALL carry id, model, choices (each with a message of role and content) and usage with prompt_tokens, completion_tokens and total_tokens.

Testing Requirements

Unit Tests

class TestOpenRouterProvider:
    async def test_fallback_chain(self):
        provider = OpenRouterProvider()
        result = await provider.call_with_fallback("coding", prompt)
        assert result.model in FALLBACK_CHAINS["coding"]
    
    async def test_cost_tracking(self):
        await provider.call_model("gpt-4o", prompt, input_tokens=100)
        cost = await provider.get_cost("gpt-4o")
        assert cost > 0

REQ-053-055: Unit tests SHALL cover the fallback chain (the returned model belongs to the agent type's chain) and cost tracking (the recorded cost of a call is greater than zero).

Integration Tests

• API connectivity to all providers
• Fallback mechanism
• Cost calculation
• Rate limiting

REQ-053-056: Integration tests SHALL cover API connectivity to all providers, the fallback mechanism, cost calculation and rate limiting.

Load Testing

• Concurrent requests: 100
• Requests per second: 50
• Model switching speed
• Cache performance

REQ-053-101: Load testing SHALL exercise 100 concurrent requests at 50 requests per second, and SHALL measure model switching speed and cache performance.

Test Coverage

• Unit: >85%
• Integration: >95%
• Provider APIs: 100%

REQ-053-102: Test coverage SHALL be greater than 85% for unit tests, greater than 95% for integration tests, and 100% for provider APIs.

Governance (PAM++)

Policy Enforcement

• Model access requires appropriate permissions
• Cost limits enforced per user/project
• API key management restricted
• Usage monitoring logged

REQ-053-057: Model access SHALL require the appropriate permissions.

REQ-053-058: Cost limits SHALL be enforced per user and per project.

REQ-053-059: API key management SHALL be restricted.

REQ-053-060: Usage monitoring SHALL be logged.

Audit Logging

async def log_model_call(model: str, tokens: int, cost: float):
    await audit_log({
        "event_type": "model_call",
        "model": model,
        "tokens": tokens,
        "cost": cost,
        "user": request.user.id,
        "timestamp": datetime.utcnow()
    })

REQ-053-061: Every model call SHALL be written to the audit log as an event of type model_call carrying the model, the token count, the cost, the requesting user and a timestamp.

Data Retention

• Cost records: 2 years
• Usage logs: 1 year
• API key history: Until revoked
• Performance metrics: 90 days

REQ-053-103: Cost records SHALL be retained for 2 years, usage logs for 1 year, API key history until revoked, and performance metrics for 90 days.

Non-Functional Requirements

Response Times

• Cached models: <100ms
• Uncached models: <2s
• Fallback switch: <500ms
• Cost calculation: <10ms

REQ-053-104: Response times SHALL be under 100ms for cached models, under 2s for uncached models, under 500ms for a fallback switch, and under 10ms for cost calculation.

Throughput

• Requests per second: 100
• Concurrent users: 50
• Models per provider: 10

REQ-053-105: The system SHALL sustain 100 requests per second, 50 concurrent users, and 10 models per provider.

Caching

• Model lists: 1 hour
• Pricing: 24 hours
• Decision cache: 1 hour

REQ-053-106: Model lists SHALL be cached for 1 hour, pricing for 24 hours, and routing decisions for 1 hour.

Monitoring

Metrics

• Request volume by model
• Cost per user/project
• Success/failure rates
• Latency percentiles
• Cache hit rates

REQ-053-062: The system SHALL emit metrics for request volume by model, cost per user and project, success and failure rates, latency percentiles, and cache hit rates.

Alerts

• Budget exceeded (90%)
• High error rate (>5%)
• Slow responses (>5s)
• Fallback chain exhausted

REQ-053-063: The system SHALL alert on budget exceeded (90%), high error rate (>5%), slow responses (>5s), and fallback chain exhausted.

Dashboards

• Real-time usage
• Cost trends
• Model performance
• Provider health

REQ-053-064: The system SHALL provide dashboards for real-time usage, cost trends, model performance, and provider health.

Security

API Key Management

• Encrypt all API keys
• Rotate quarterly
• Monitor usage anomalies
• Revoke compromised keys

REQ-053-065: The system SHALL encrypt all API keys, rotate them quarterly, monitor usage anomalies, and revoke compromised keys.

Request Validation

• Validate model names
• Limit context length
• Sanitize prompts
• Check budget before request

REQ-053-066: The system SHALL validate model names, limit context length, sanitize prompts, and check the budget before issuing a request.

Privacy

• Don't log prompts
• Anonymize usage data
• Comply with data retention
• User consent for data processing

REQ-053-067: The system SHALL NOT log prompts, SHALL anonymize usage data, SHALL comply with the data-retention policy, and SHALL require user consent for data processing.

Technical Design

Architecture Components

┌─────────────────────────────────────┐
│   OpenRouter Provider               │
├─────────────────────────────────────┤
│                                     │
│  ┌─────────────┐  ┌──────────────┐ │
│  │ Model Router│  │ Fallback     │ │
│  └─────────────┘  └──────────────┘ │
│                                     │
│  ┌─────────────┐  ┌──────────────┐ │
│  │ Cost Tracker│  │ Budget       │ │
│  └─────────────┘  └──────────────┘ │
│                                     │
│  ┌─────────────┐  ┌──────────────┐ │
│  │ Retry Logic │  │ Cache Layer  │ │
│  └─────────────┘  └──────────────┘ │
└─────────────────────────────────────┘
         │              │
    ┌────▼────┐    ┌────▼────┐
    │ Provider│    │   API   │
    │  APIs   │    │  Layer  │
    └─────────┘    └─────────┘

REQ-053-068: The OpenRouter provider SHALL be composed of a Model Router, a Fallback component, a Cost Tracker, a Budget component, Retry Logic and a Cache Layer, sitting above the provider APIs and the API layer.

Data Flow

1. Request received with agent type and prompt
2. Router selects model based on agent type configuration
3. Cost budget checked against user/project limits
4. API call initiated with retry logic
5. Success: Cost recorded, response returned
6. Failure: Fallback model tried per chain
7. All fallbacks exhausted: Error returned with partial response

REQ-053-069: A request SHALL be processed in this order: receive the request with its agent type and prompt; select the model from the agent type configuration; check the cost budget against the user and project limits; issue the API call under retry logic; on success record the cost and return the response; on failure try the fallback model per chain; and when all fallbacks are exhausted return an error with a partial response.

Examples

Example 1: Basic Model Call

# Request
request = {
    "model": "claude-3-5-sonnet",
    "messages": [
        {"role": "user", "content": "Write a Python function"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000
}

# Response
{
    "id": "gen-123",
    "model": "claude-3-5-sonnet",
    "choices": [
        {"message": {"role": "assistant", "content": "def hello(): ..."}}
    ],
    "usage": {
        "prompt_tokens": 15,
        "completion_tokens": 25,
        "total_tokens": 40,
        "cost": 0.00032
    }
}

Example 2: Fallback Scenario

# Primary model fails
call_model("claude-3-5-sonnet", prompt)
# ModelError: rate_limit_exceeded

# Automatic fallback to next in chain
call_model("claude-3-opus", prompt)
# Success

# Log fallback
{
    "event": "fallback_used",
    "from_model": "claude-3-5-sonnet",
    "to_model": "claude-3-opus",
    "reason": "rate_limit_exceeded"
}

Example 3: Budget Alert

# User approaching budget
current_spend = 80.0
daily_limit = 100.0
threshold = 80.0  # 80% of limit

# Alert triggered
notify_user("user123", {
    "type": "budget_threshold",
    "current_spend": 80.0,
    "daily_limit": 100.0,
    "percentage": 80,
    "message": "You've used 80% of your daily budget"
})

Integration Points

Provider APIs

• Anthropic Claude: Messages API v1 for text generation
• OpenAI: Chat Completions API for model interaction
• Google Gemini: Generative Language API for multimodal models
• Ollama: Local REST API for on-premise models

REQ-053-070: The system SHALL integrate with the Anthropic Claude Messages API v1 for text generation, the OpenAI Chat Completions API for model interaction, the Google Gemini Generative Language API for multimodal models, and the Ollama local REST API for on-premise models.

Internal Services

• Cost Service: PostgreSQL database for usage tracking
• Cache Layer: Redis for model lists, pricing, and decisions
• Message Queue: RabbitMQ for async processing and alerts
• Audit Service: Centralized logging with ELK stack

REQ-053-071: The system SHALL use a PostgreSQL-backed cost service for usage tracking, Redis as the cache layer for model lists, pricing and decisions, RabbitMQ as the message queue for async processing and alerts, and a centralized ELK-stack audit service.

External Integrations

• Payment Provider: Stripe for budget management
• Monitoring: Prometheus for metrics, Grafana for dashboards
• Notification: Email/SMS for budget and error alerts
• API Gateway: Kong for rate limiting and authentication

REQ-053-072: The system SHALL integrate with Stripe as the payment provider for budget management, Prometheus for metrics and Grafana for dashboards, email/SMS notification for budget and error alerts, and the Kong API gateway for rate limiting and authentication.

Success Criteria

Functional Criteria

• [ ] All 4 providers (Anthropic, OpenAI, Google, Ollama) operational
• [ ] Fallback chains tested and working for all agent types
• [ ] Cost tracking accurate within 0.01% of provider billing
• [ ] Budget enforcement blocking requests at configured limits
• [ ] Retry logic reducing failure rate by >90%

REQ-053-073: All 4 providers (Anthropic, OpenAI, Google, Ollama) SHALL be operational.

REQ-053-074: Fallback chains SHALL be tested and working for all agent types.

REQ-053-075: Cost tracking SHALL be accurate within 0.01% of provider billing.

REQ-053-076: Budget enforcement SHALL block requests at the configured limits.

REQ-053-077: Retry logic SHALL reduce the failure rate by more than 90%.

Performance Criteria

• [ ] 95th percentile response time <2s for uncached models
• [ ] Cache hit rate >70% for model lists and routing decisions
• [ ] Zero budget overruns across all users and projects
• [ ] Fallback chain execution <500ms average
• [ ] Support for 100 concurrent requests per second

REQ-053-107: The 95th-percentile response time SHALL be under 2s for uncached models.

REQ-053-108: The cache hit rate SHALL exceed 70% for model lists and routing decisions.

REQ-053-109: There SHALL be zero budget overruns across all users and projects.

REQ-053-110: Fallback chain execution SHALL average under 500ms.

REQ-053-111: The system SHALL support 100 concurrent requests per second.

Quality Criteria

• [ ] Test coverage >85% unit, >95% integration
• [ ] Zero critical security vulnerabilities
• [ ] Audit log completeness 100% for compliance
• [ ] Documentation coverage 100% for all API endpoints
• [ ] Zero data retention policy violations

REQ-053-112: Test coverage SHALL exceed 85% unit and 95% integration.

REQ-053-113: There SHALL be zero critical security vulnerabilities.

REQ-053-114: Audit log completeness SHALL be 100% for compliance.

REQ-053-115: Documentation coverage SHALL be 100% for all API endpoints.

REQ-053-116: There SHALL be zero data-retention policy violations.

Business Criteria

• [ ] Cost optimization achieving 20%+ savings vs single provider
• [ ] System availability >99.9% uptime
• [ ] User satisfaction score >4.5/5 for reliability
• [ ] Provider diversity reducing vendor lock-in
• [ ] Admin time reduction >50% for model management

REQ-053-117: Cost optimization SHALL achieve savings of 20% or more against a single provider.

REQ-053-118: System availability SHALL exceed 99.9% uptime.

REQ-053-119: The user satisfaction score for reliability SHALL exceed 4.5/5.

REQ-053-120: Provider diversity SHALL reduce vendor lock-in.

REQ-053-121: Admin time for model management SHALL be reduced by more than 50%.

Revision History

Related

• RTM page for 053-openrouter-provider
• All specs
• RTM overview
• Gates