Orchestrata Ultimate › Plans › spec-audit › D-providers-llm

Cluster D — Providers & LLM — spec audit + implementation plan

Source: plans/spec-audit/D-providers-llm.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-04 · 234 lines.

Prose pending.

Plan text

Cluster D — Providers & LLM — spec audit + implementation plan

Investigation agent D. READ-ONLY audit of src/orchestrata_ultimate/ against 4 specs, evidence-based, per _METHOD.md. Prior work confirmed/updated against docs/spec-gap-register.md (which audited SPEC-011 and SPEC-028).

Specs: SPEC-011 Provider Manager, SPEC-025 Code-Gen Drivers, SPEC-028 LLM Client, SPEC-053 OpenRouter.

SPEC-011 Provider Manager — PARTIAL

• Specifies: unified LLM provider abstraction — routing, fallback, circuit breaker, rate limiting, cost tracking, budget enforcement, PAM++ governance, audit trail, events.
• Verified:
  • ✅ IProviderManager implemented and wired — ProviderManager (providers/provider_manager.py:82) is used by the live LLM facade (refs ProviderManager → 8 uses, all in llm/client.py). execute/estimate_cost/get_usage/register/deregister/get_remaining_budget/cancel/get_provider_health all present (provider_manager.py:196-523). Component lifecycle initialize/start/stop/get_health ✅.
  • ✅ FR-015 token counting — REMEDIATED (confirms register). Flat chars/4 heuristic replaced by the shared TiktokenCounter: _token_counter_for (provider_manager.py:531, memoised per model, OpenAI-family→exact encoding, others→cl100k_base/gpt-4) feeding _estimate_cost_for_pricing (:553). Shares SPEC-010's memory/token_counter.py:44 (not a third heuristic). Wired into _enforce_budget (:314) and estimate_cost (:451).
  • ✅ FR-003 routing / FR-008 fallback chain / FR-009 circuit breaker / FR-010 rate limiting — delegated to router.py, circuit_breaker.py, rate_limiter.py; fallback loop in execute (:274-311).
  • ✅ FR-011 cost tracking + FR-012 budget enforcement — cost_tracker.py: TOCTOU-safe Lua budget check (:33), Redis aggregation by volk/session/project/provider/model (record_cost :108).
  • ✅ FR-007 adapters — OpenAI/Anthropic/DeepSeek/Ollama/OpenRouter/Claude-CLI all under providers/adapters/ (7 adapters).
  • ❌ FR-013 PAM++ Prolog governance — ABSENT. No allowed_model / within_budget / can_use_provider / model allowlist-denylist predicates anywhere; execute() performs no model-access governance check (grep for these predicates across src/ = 0 hits). The spec's entire "Prolog Governance Predicates" section is unimplemented.
  • ❌ FR-014 Neo4j audit trail — ABSENT. cost_tracker is Redis-only; no (:ProviderAudit) / (:CostRecord) / (:ProviderHealth) nodes, no cold migration on session.closed (grep ProviderAudit|neo4j in providers/ = 0). Audit is hot-Redis only.
  • ❌ Event Delivery — ABSENT. No kernel.emit_event / EventBus emission of the spec's topics (provider.request.completed, provider.fallback.triggered, provider.budget.*, …); no reactive subscriptions to volk.lifecycle.terminated / session.closed (grep for topics = 0). Budget warning events (80/90/95%) from FR-012 also not emitted.
  • ⚠️ FR-005 config source — DIVERGENT. Providers are configured in code in llm/client._register_* reading env keys directly (ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, OPENROUTER_API_KEY, OLLAMA_BASE_URL), not from configs/production.yaml with ${API_KEY} interpolation. Functional but a different mechanism than the spec's YAML config.
• Remaining work / plan:
  • Scope: (a) FR-013 Prolog model-access + budget governance wired into execute() before dispatch; (b) FR-014 Neo4j audit persistence + hot→cold migration on session close; (c) EventBus emission of provider lifecycle/budget events + reactive subscriptions. FR-005 YAML config is a smaller divergence (either adopt production.yaml loading or update the spec to the code-driven registration).
  • Target files: providers/provider_manager.py (governance hook in execute/_enforce_budget, event emission points), providers/cost_tracker.py (Neo4j cold-store + session-close migration), a new providers/governance.py (janus predicate calls), governance .pl rule additions; wiring to the kernel EventBus.
  • Approach: Add a governance gate calling the janus adapter (ground facts asserted, variable rules in .pl) in execute(); emit SystemEvents via kernel.emit_event at completion/failure/fallback/ budget thresholds; persist ProviderAudit/CostRecord to Neo4j and subscribe migrate_to_cold-style logic to session.closed.
  • Dependencies: governance/Prolog-janus subsystem (SPEC-013 PAM++), kernel EventBus (SPEC-001), Neo4j persistence + session-close event — the same unwired session.closed EventBus decision that blocks SPEC-010 FR-008 (memory cold migration). Governance + audit are shared design decisions, not point fixes.
  • Effort: L. Risk: med — governance-in-hot-path adds latency (NFR-001 <5ms) and couples to the janus adapter + EventBus lifecycle; audit persistence is additive but needs the persistent-engine wiring.
• Evidence log:
  • symbols/refs ProviderManager → def provider_manager.py:82, 8 uses in llm/client.py (wired).
  • Read provider_manager.py (full): FR-015 counters at :531/:553; execute :262-311 has no governance call and no event emission; initialize :127 subscribes to nothing.
  • Read cost_tracker.py (full): Redis-only, no Neo4j.
  • grep -riE 'emit_event|event_bus|neo4j|prolog|governance|allowed_model|within_budget|can_use_provider' over providers/ → 0 hits; same predicates over all src/ → 0 hits; provider\.(request|fallback|budget|...) topics over src/ → 0 hits.

SPEC-025 Code Generation Drivers — DONE (with an FR-006 divergence)

• Specifies: a uniform async AsyncCodeDriver Protocol over 7 code-gen backends + a provider-routing factory get_driver + shared FILE:/markdown extraction utilities.
• Verified:
  • ✅ FR-001/FR-012 AsyncCodeDriver Protocol, @runtime_checkable, execute()->tuple[bool,str] (drivers/base.py:8,20,32).
  • ✅ All 7 drivers present: ClaudeCodeDriver, ClaudeAPIDriver, AiderDriver, DeepSeekCoderDriver, QwenDriver, BrowserDriver (×3 providers via BROWSER_PROVIDERS). (symbols Driver; browser_providers.py:91).
  • ✅ FR-002/FR-011 factory routing + unknown→ClaudeAPIDriver fallback with warning (driver_factory.py:44-85); routing table matches spec (claude/anthropic, deepseek, aider, qwen/ollama, *_browser). Wired — refs get_driver → used by swarm coordinator, codedirector, junior (coordinator.py:391, codedirector/volk.py:343, junior/volk.py:201).
  • ✅ FR-010 extract_file_blocks / extract_markdown_code_blocks (drivers/extraction.py:24,70).
  • ✅ FR-005 ClaudeAPIDriver workspace scan; FR-004 asyncio.wait_for timeouts; FR-007/008/009 browser session persistence + 3-tier recovery + rate limit present (browser_driver.py methods _recover_session, _save_storage_state, _enforce_rate_limit, _is_session_expired).
  • ⚠️ FR-006 DeepSeek direct-API — DIVERGENT. Spec: "DeepSeekCoderDriver MUST call the DeepSeek HTTP API with model=deepseek-coder and use DEEPSEEK_API_KEY." Actual: it routes through call_llm(CODER_SYSTEM_PROMPT, prompt, …) (deepseek_coder_driver.py:272,275) → SPEC-028 facade → ProviderManager. No direct DeepSeek HTTP call, no model=deepseek-coder. Intentional consolidation (all LLM traffic via the one facade); the driver is also richer than spec (syntax validate, ruff autofix, missing-file recovery). ERR-010/ERR-011 (direct-API error strings) no longer apply verbatim.
  • Note: @traces already reference spec="025" (the spec's "next pass will update decorators to 025" is done).
• Remaining work / plan: (spec-only — no code fix)
  • Scope: Update SPEC-025 FR-006 + the DeepSeekCoderDriver section + ERR-010/011 to state the driver routes through call_llm (SPEC-028), not the DeepSeek HTTP API directly. Optionally note the extra validation/recovery behaviour.
  • Target files: specs/025-code-generation-drivers/spec.md (no src/ change).
  • Dependencies: SPEC-028 (call_llm) — the divergence exists because of the facade consolidation.
  • Effort: S. Risk: low (documentation reconciliation).
• Evidence log: Read base.py, driver_factory.py, deepseek_coder_driver.py (full); symbols Driver (7 classes), symbols extract_ (extraction.py:24/70), grep BROWSER_PROVIDERS (browser_providers.py:91), refs get_driver (3 consumers).

SPEC-028 LLM Client Facade — DIVERGENCE (core DONE; API redesigned)

• Specifies: single async call_llm() entry + call_reasoner(), per-volk VolkConfig with env overrides, resolve_provider/resolve_model, transient-retry, KG injection + compliance validation, lazy ProviderManager singleton, LLMClientError/LLMProviderUnavailableError.
• Verified:
  • ✅ FR-001/FR-002 single entry call_llm + call_reasoner (llm/client.py:502,653); heavily wired (refs call_llm → 23 uses). All other functions delegate to call_llm.
  • ✅ FR-005/FR-006 transient retry + permanent propagation — _execute_with_retry (:373), _is_transient marker set (:53), MAX_RETRIES=6, RETRY_DELAYS=[5,10,20,40,60,60] (:50). (Spec default is 5 — minor numeric divergence, tuned to clear deepseek 429 windows.)
  • ✅ FR-007 KG injection (graph_context prepended into the prompt tokens, :546) + ✅ FR-008 compliance validation via GraphContextValidator with MAX_VALIDATION_RETRIES retry loop (:575-635).
  • ✅ FR-004/FR-012/FR-013 lazy ProviderManager singleton + adapter registration (_get_provider_manager :312, _register_adapters :294 → claude_cli, anthropic, deepseek, openrouter, ollama — the spec's 5 providers).
  • ✅ FR-010 exception hierarchy LLMClientError / LLMProviderUnavailableError (llm/exceptions.py).
  • 🔻 FR-009 env overrides — DIVERGENCE (confirms register). Live path llm/config.py:1-8 states outright "single source of truth is spike-profiles.yaml. No env variable overrides." Profile chosen by SPIKE_PROFILE (:151). Extra finding: a parallel config core/infra_config.py:203-219 DOES honor ORCHESTRATA_LLM_PROVIDER/_MODEL/_REASONING_MODEL, and exposes its own get_volk_provider/get_volk_model — but the LLM client and driver factory import orchestrata_ultimate.llm.config, not infra_config (client.py:33, driver_factory.py:60), so the env-override path is effectively orphaned/duplicate for LLM routing. (Also inconsistent: infra_config _REASONING_VOLKS={U:Ar} vs llm/config REASONING_AGENTS={U:Ar,U:SD}.)
  • ❌ FR-011 resolve_provider() / resolve_model() — ABSENT in the facade (grep 'def resolve_' llm/ = none; symbols resolve_provider = 0; the one resolve_model hit is an unrelated registry.py:83 method). Replaced by get_volk_provider/get_volk_model.
  • 🔻 Signature/return DIVERGENCE. Spec's call_llm(volk_alias, messages, kg_context, *overrides) -> LLMResponse; actual call_llm(system_prompt, user_message, model, temperature, volk, graph_context) -> str (:502). The str return actually matches the spec's own US-028-001 ("Return type is a plain str") — the spec is internally inconsistent (LLMResponse data model vs US-028-001). No LLMResponse/LLMRequest objects, no provider_override/reasoning_model_override/ max_retries_override/metadata params, no per-Volk ORCHESTRATA_LLM_MODEL_U:AR override.
• Remaining work / plan: (spec update + a duplication cleanup decision — no functional hole)
  • Scope: Rewrite SPEC-028 to the profiles-based redesign: SPIKE_PROFILE-selected spike-profiles.yaml as single source of truth (no env overrides), str-returning call_llm(system_prompt,user_message,…), get_volk_provider/get_volk_model in place of resolve_provider/resolve_model, MAX_RETRIES=6. Separately: decide the fate of core/infra_config.py's duplicate LLM config (env-honoring but orphaned) — retire it or make it the deliberate CLI/test override layer and wire it. That is a consolidation call, not a fix.
  • Target files: specs/028-llm-client/spec.md; decision on core/infra_config.py (possible removal of its LLM-routing block) — no change to llm/client.py/llm/config.py (they are the intended design).
  • Dependencies: SPEC-011 (ProviderManager), SPEC-025 (driver_factory reads llm.config), and the core/infra_config duplication (touches many env-config sites; see the "read env at call time" note).
  • Effort: S (spec) / M (if infra_config consolidation is taken). Risk: low/med — the duplicate-config removal must not break CLI/test env-override expectations.
• Evidence log: Read llm/client.py + llm/config.py (full); refs call_reasoner (facade client.py:653); symbols resolve_provider=0, resolve_model=registry only; grep ORCHESTRATA_LLM_ → only core/infra_config.py:203-211; read infra_config.py:195-219 (env-override block present but unused by the LLM client); llm/__init__.py exports.

SPEC-053 OpenRouter Provider — DIVERGENCE (realized as an adapter under SPEC-011/028)

• Specifies: an OpenRouterProvider (chat_completion, get_available_models, get_model_cost) owning multi-provider access, FALLBACK_CHAINS, RETRY_CONFIG, MODEL_PRICING/calculate_cost, budget enforcement/alerts, caching, audit — a monolithic provider subsystem.
• Verified:
  • ❌ No standalone OpenRouterProvider class, no chat_completion / get_available_models / get_model_cost (symbols OpenRouterProvider = 0). The class in the SPEC-028 class-diagram is likewise not built.
  • ✅ OpenRouter is implemented as one adapter in the unified framework: OpenRouterAdapter(BaseAdapter) implementing IProvider.complete/get_models/supports_capability (providers/adapters/openrouter_adapter.py:27), OpenAI-compatible endpoint + HTTP-Referer/X-Title headers. Wired (not orphaned): registered in llm/client._register_openrouter (:215, used :260) with 13 upstream models (Claude/Qwen/Kimi/DeepSeek/Llama/Gemini/MiniMax); reachable via the openrouter provider in configs/spike-profiles.yaml (kimi profile) and PROVIDER_MODELS (llm/config.py:50).
  • 🔻 The spec's other responsibilities are decomposed into SPEC-011 + SPEC-028, not this provider: fallback chain + circuit breaker + cost tracking + budget → SPEC-011 (router.py, circuit_breaker.py, cost_tracker.py); retry/backoff → SPEC-028 (_execute_with_retry). So FR-002/003/004/005/007 exist system-wide but not as an OpenRouter-owned feature. Per-agent model routing = spike-profiles.yaml, not FALLBACK_CHAINS.
  • ❌ FR-011 caching (model list 1h / pricing 24h / routing 1h) — genuinely absent everywhere (minor perf feature, unwired).
• Remaining work / plan: (spec reconciliation — no real code work)
  • Scope: Rewrite/retire SPEC-053 to reflect reality: "OpenRouter is an IProvider adapter (OpenRouterAdapter) plugged into the SPEC-011 ProviderManager; routing/fallback/cost/budget/retry are owned by SPEC-011 + SPEC-028, not a standalone OpenRouter subsystem." Drop the OpenRouterProvider class contract, FALLBACK_CHAINS/MODEL_PRICING/RETRY_CONFIG duplication, and the Postgres/RabbitMQ/Stripe/Kong integration points (not this codebase). Optionally implement the model-list/pricing caching if the perf target is wanted.
  • Target files: specs/053-openrouter-provider/spec.md (mark superseded-by-011+028); no src/ change.
  • Dependencies: SPEC-011 + SPEC-028 (reconcile the three specs together — 053 overlaps both).
  • Effort: S (spec) — or S/M if the caching feature is added to the adapter/registry. Risk: low.
• Evidence log: symbols OpenRouterProvider=0; read openrouter_adapter.py (full); refs OpenRouterAdapter → registered/used in llm/client.py:233,260; configs/spike-profiles.yaml kimi profile uses provider: openrouter; PROVIDER_MODELS['openrouter'] in llm/config.py:50.

Cluster summary table

Notes / non-gaps (do NOT "fix")

• memory/token_counter.py:59 except ImportError, KeyError: and deepseek_coder_driver.py:503 except OSError, UnicodeDecodeError: are valid PEP 758 Python-3.14 parenthesis-free except — not bugs.
• SPEC-011 FR-015 (chars/4 → TiktokenCounter) was remediated in a prior session; confirmed still wired.

Cross-cluster dependencies noticed

• SPEC-011 ↔ SPEC-010: both need the same unresolved session.closed → EventBus/persistent-engine wiring (SPEC-011 Neo4j cost cold-migration; SPEC-010 FR-008 memory migrate_to_cold). Solve once.
• SPEC-011 ↔ governance (SPEC-013 PAM++/janus): FR-013 model-access + budget-override predicates depend on the Prolog-janus adapter (ground facts asserted, variable rules consulted from .pl).
• SPEC-011 ↔ SPEC-001 EventBus: provider event emission depends on kernel.emit_event.
• SPEC-028 ↔ core/infra_config: a duplicate get_volk_provider/get_volk_model (env-honoring but orphaned) coexists with the live llm/config.py; consolidation is a design decision (relates to the "read env at call time" memory note). Touches SPEC-025 (driver_factory imports llm.config).
• SPEC-053 ⊂ SPEC-011 + SPEC-028: SPEC-053 largely restates capabilities that live in 011/028; reconcile the three specs in one pass rather than treating 053 as independent work.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks