Orchestrata Ultimate › Plans › spec-audit › reverify-2026-10-18 › D-providers-llm

Cluster D — Providers & LLM — re-verification (2026-10-18)

Source: plans/spec-audit/reverify-2026-10-18/D-providers-llm.md at 8a2cb42ad · last changed 2026-10-18 · first committed 2026-10-18 · 284 lines.

Prose pending.

Plan text

Cluster D — Providers & LLM — re-verification (2026-10-18)

Re-verification of src/orchestrata_ultimate/ against 4 specs on branch components (337 commits ahead of the 2026-07-14 audits). Method per _METHOD.md + the re-verify brief: AST toolkit only for Python structure; .pl Prolog grep for governance predicates; .md specs read directly. Prior work confirmed/updated against docs/spec-gap-register.md and plans/spec-audit/D-providers-llm.md (session 195).

Trusted main-session verdicts (NOT re-verified): SPEC-028 FR-009 env-override divergence RESOLVED (spec updated); SPEC-011 FR-015 real TiktokenCounter landed. Both re-confirmed incidentally as present (see below) but not re-litigated.

SPEC-011 Provider Manager — PARTIAL

• Specifies: unified LLM provider abstraction — routing, fallback, circuit breaker, rate limiting, cost tracking, budget enforcement, PAM++ governance, Neo4j audit trail, EventBus events.

• Verified:

  • ✅ FR-001 interface + lifecycle. ProviderManagerProtocol (core/protocols/providers.py:241, inherits ComponentProtocol), ProviderManager (providers/provider_manager.py:75) implements it; initialize/start/stop/get_health (:123/:143/:150/:175); ComponentMetadata carries component_type=ComponentType.PROVIDER_MANAGER (:104). Wired — refs ProviderManager → 13 USE sites in llm/client.py (e.g. :371/:405/:482/:504). ⚠️ DI-registration is a lazy module-level singleton in llm/client.py:482 (_get_provider_manager), NOT a DIContainer resolution — consistent with the D1 orphan-DI-container decision, not a functional hole.
  • ✅ FR-002 execute / FR-004 estimate / FR-016 cancel. execute (:258), estimate_cost (:453), get_usage (:472), get_remaining_budget (:489), cancel (:502), get_provider_health (:512). _invoke_provider (:348) registers the in-flight task (self._in_flight[request.request_id] = complete_task, :361) so cancel/stop can interrupt it — FR-016 is wired, not a stub.
  • ✅ FR-003 routing / FR-008 fallback / FR-009 breaker / FR-010 limiter. RequestRouter.route/get_fallback_chain (router.py:55/:88); fallback loop in execute (:274-307); CircuitBreaker (circuit_breaker.py:47, defaults failure_threshold=5, cooldown_seconds=60, success_threshold=3 — matches spec FR-009); RateLimiter (rate_limiter.py:25, rpm_limit=60, tpm_limit=100000, headroom_pct/has_low_headroom).
  • ✅ FR-006 model registry. ProviderRegistry (registry.py:22) with resolve_model (alias, :80), find_by_capability (:85), get_pricing (:72), model_count (:108).
  • ✅ FR-007 adapters. ProviderProtocol (core/protocols/providers.py:217, complete/get_models/ supports_capability); BaseAdapter (adapters/base.py:44); 7 concrete adapters — anthropic, claude_cli, deepseek, ollama, openai, openrouter (all under providers/adapters/).
  • ✅ FR-011 cost tracking / FR-012 budget enforcement. CostTracker (cost_tracker.py:71, Redis-backed): calculate_cost (:95), record_cost (:128, aggregates volk/session/project + canonical budget key), check_budget (:106, TOCTOU-safe Lua evalsha), get_usage (:186), get_remaining_budget (:217). Enforcement wired via _enforce_budget (provider_manager.py:312) before dispatch.
  • ✅ US-007 health monitoring. HealthMonitor (health_monitor.py:28, sliding-window latency/error/percentiles, _percentile :98), fed by _record_success/_record_failure (provider_manager.py:370/:416).
  • ✅ FR-015 token counting (trusted, re-confirmed present): _token_counter_for (provider_manager.py:532, memoised TiktokenCounter, OpenAI-family→exact encoding, others→gpt-4/cl100k_base) → _estimate_cost_for_pricing (:554) → estimate_cost/_enforce_budget.
  • ⚠️ FR-005 config source — DIVERGENT. Providers are registered in code by llm/client.py _register_claude_cli/_register_anthropic/_register_deepseek/_register_openai/_register_openrouter/ _register_ollama (:404-446) reading env keys, not from configs/production.yaml with ${API_KEY} interpolation. Functionally complete; mechanism differs from spec.
  • ❌ FR-013 PAM++ Prolog governance — ABSENT. execute() (:258-311) performs no model-access governance check; provider_manager.py imports no governance/janus module. grep -rn 'allowed_model|within_budget|can_use_provider|budget_override|model_allowlist|model_denylist|volk_budget_limit' src/orchestrata_ultimate/prolog/ → 0 hits (all 33 .pl files). No allowlist/denylist/budget predicates exist.
  • ❌ FR-014 Neo4j audit trail — ABSENT. CostTracker is Redis-only (cost_tracker.py:1-241, prefix orchestrata:provider:cost); no Neo4j import, no (:ProviderAudit)/(:CostRecord)/(:ProviderHealth) nodes, no cold migration. symbols ProviderAudit|ProviderHealth → only core/protocols/providers.py:58 CostRecord (the dataclass).
  • ❌ Event delivery — ABSENT. initialize (:123-140) subscribes to nothing; no kernel.emit_event / event_bus.subscribe import anywhere in providers/; none of the spec's 9 topics (provider.request.completed, provider.fallback.triggered, provider.budget.warning, …) is emitted. FR-012 budget-warning events (80/90/95%) also not emitted.

• Remaining work / plan:

  • Scope: (a) FR-013 Prolog model-access + budget-governance gate in execute() before dispatch; (b) FR-014 Neo4j ProviderAudit/CostRecord/ProviderHealth persistence + hot→cold migration on session.closed; (c) EventBus emission of provider lifecycle/budget/fallback events + reactive subscriptions to volk.lifecycle.terminated/session.closed. FR-005 YAML config is a smaller divergence (adopt production.yaml loading or update the spec to code-driven registration).
  • Target files: providers/provider_manager.py (governance hook in execute/_enforce_budget; event emission); providers/cost_tracker.py (Neo4j cold-store + session-close migration); a new providers/governance.py (janus predicate calls); governance .pl rule additions; EventBus wiring.
  • Approach: add a governance gate calling the janus adapter (ground facts asserted, variable rules in .pl) in execute(); emit SystemEvents via kernel.emit_event at completion/failure/fallback/budget thresholds; persist audit nodes to Neo4j and subscribe a migrate_to_cold-style hook to session.closed.
  • Dependencies: Prolog-janus governance (SPEC-013 PAM++), kernel EventBus (SPEC-001), Neo4j persistence + session.closed wiring — the same unwired session.closed decision that blocks SPEC-010 FR-008.
  • Effort: L. Risk: med — governance-in-hot-path couples to janus + EventBus lifecycle and adds latency (NFR-001 <5ms); audit persistence is additive but needs the persistent-engine wiring.

• Evidence log:

  • symbols ProviderManager → class provider_manager.py:75; refs ProviderManager → 13 USE in llm/client.py.
  • source providers.provider_manager (full read): execute :258, no governance call, no event emission; initialize :123 subscribes to nothing; _invoke_provider :361 tracks _in_flight.
  • source providers.cost_tracker → Redis-only, Lua budget script, no Neo4j.
  • symbols CircuitBreaker|RateLimiter|RequestRouter|HealthMonitor|ProviderRegistry|CostTracker → all present.
  • grep -rn 'allowed_model|within_budget|can_use_provider|budget_override|...' src/orchestrata_ultimate/prolog/ → 0 hits.
  • symbols ProviderAudit|ProviderHealth → 0 in src/ (only the CostRecord dataclass at protocols/providers.py:58).

SPEC-025 Code Generation Drivers — DONE (FR-006 divergence)

• Specifies: a uniform async AsyncCodeDriver Protocol over 7 code-gen backends + a provider-routing factory get_driver + shared FILE:/markdown extraction utilities.

• Verified:

  • ✅ FR-001 / FR-012 Protocol. AsyncCodeDriver (drivers/base.py:20) is @runtime_checkable (:19) with execute(task, files, is_fix) -> tuple[bool, str] (:32). Traces now spec="025" (the spec's "next pass updates decorators to 025" is done).
  • ✅ 7 drivers present. ClaudeCodeDriver (claude_code_driver.py:119), ClaudeAPIDriver (claude_api_driver.py:98), AiderDriver (aider_driver.py:77), DeepSeekCoderDriver (deepseek_coder_driver.py:276), QwenDriver (qwen_driver.py:112), BrowserDriver (browser_driver.py:34), BrowserProviderConfig (browser_providers.py:20).
  • ✅ FR-002 factory + FR-011 fallback. get_driver (driver_factory.py:46) reads get_volk_provider from orchestrata_ultimate.llm.config (:72); routing table claude/anthropic→ClaudeAPIDriver, deepseek/openrouter→DeepSeekCoderDriver, aider→AiderDriver, qwen/ollama→QwenDriver, *_browser→BrowserDriver, claude_code→ClaudeCodeDriver, unknown→ClaudeAPIDriver with warning (:99-105). Wired — refs get_driver → agents/swarm/coordinator.py:417, agents/volks/codedirector/volk.py:502, agents/volks/junior/volk.py:215.
  • ✅ FR-010 extraction. extract_file_blocks (extraction.py:21), extract_markdown_code_blocks (:63).
  • ✅ FR-003/FR-004 CLI + timeout. each CLI driver locates binary via shutil.which and wraps subprocess in asyncio.wait_for (e.g. aider_driver._run_process:173, claude_code_driver._run_process:361).
  • ✅ FR-005 workspace scan. ClaudeAPIDriver scans post-execution for source files (claude_api_driver.py:135+).
  • ✅ FR-007/FR-008/FR-009 browser. BrowserDriver session persistence _save_storage_state (:402), 3-tier recovery _recover_session (:374), rate limit _enforce_rate_limit (:208), expiry _is_session_expired (:196).
  • ⚠️ FR-006 DeepSeek direct-API — DIVERGENT. DeepSeekCoderDriver.execute (deepseek_coder_driver.py:299) routes through call_llm(CODER_SYSTEM_PROMPT, prompt, temperature=0.2, volk="J:Junior") (:321-325) — no direct DeepSeek HTTP call, no model=deepseek-coder, no DEEPSEEK_API_KEY. Class docstring "using DeepSeek API directly" (:277) is stale. Driver is richer than spec (_retry_on_empty, _recover_missing_files, _autofix_lint, _validate_written_files). Fresh detail (2026-10-18): openrouter now ALSO routes to DeepSeekCoderDriver (driver_factory.py:74-85, with an explanatory comment) — prior audit listed only deepseek.

• Remaining work / plan: (spec-only, no code change)

  • Scope: update SPEC-025 FR-006 + DeepSeekCoderDriver section + ERR-010/011 to state the driver routes via call_llm (SPEC-028) and is provider-agnostic (also serves openrouter), not a direct DeepSeek HTTP client.
  • Target files: specs/025-code-generation-drivers/spec.md.
  • Approach: rewrite the two divergent paragraphs; optionally note the extra validation/recovery behavior.
  • Dependencies: SPEC-028 (call_llm) — divergence exists because of the facade consolidation.
  • Effort: S. Risk: low (documentation reconciliation).

• Evidence log:

  • symbols AsyncCodeDriver|get_driver|extract_file_blocks|extract_markdown_code_blocks → all present.
  • symbols ClaudeCodeDriver|ClaudeAPIDriver|AiderDriver|DeepSeekCoderDriver|QwenDriver|BrowserDriver|BrowserProviderConfig → 7 present.
  • source drivers.base → @runtime_checkable at :19.
  • source drivers.driver_factory → routing table incl. openrouter branch :74-85.
  • source drivers.deepseek_coder_driver → call_llm(...volk="J:Junior") at :321-325; no HTTP/DEEPSEEK_API_KEY.
  • refs get_driver → 3 production consumers.

SPEC-028 LLM Client Facade — DIVERGENCE (core DONE; API redesigned)

• Specifies: single async call_llm() entry + call_reasoner(), per-volk VolkConfig with env overrides, resolve_provider/resolve_model, transient retry, KG injection + compliance validation, lazy ProviderManager singleton, LLMClientError/LLMProviderUnavailableError.

• Verified:

  • ✅ FR-001/FR-002 single entry. call_llm (llm/client.py:819), call_reasoner (:984, delegates to call_llm with reasoning tier). Wired — refs call_llm → 11 production USE sites (via agents/volks/base.py:243 _call_llm wrapper and direct callers).
  • ✅ FR-005/FR-006 retry. MAX_RETRIES = 6, RETRY_DELAYS = [5,10,20,40,60,60] (:52-53); _is_transient (:519, _TRANSIENT_MARKERS incl. 429/500/502/503/timeout/connection-refused) + _is_rate_limit_error (:539); _execute_with_retry (:570, backoff loop + 900s total budget). Spec default is 5 — numeric tuning for deepseek 429 windows.
  • ✅ FR-007 KG injection. graph_context prepended to user_message in call_llm (:852-857) with a ## Entities used compliance prompt appended to system_prompt.
  • ✅ FR-008 compliance validation. GraphContextValidator (knowledge_graph/graph_context_validator.py:87)
    • MAX_VALIDATION_RETRIES loop (client.py:892-960); returns non-compliant response with error log after retries.
  • ✅ FR-004/FR-012/FR-013 adapter registration + lazy singleton. _register_adapters (:461), _register_* for claude_cli/anthropic/deepseek/openai/openrouter/ollama (:404-446) — 6 providers (spec lists 5; openai is an extra); _get_provider_manager lazy singleton with lock (:482).
  • ✅ FR-010 exceptions. LLMClientError (llm/exceptions.py:8), LLMProviderUnavailableError (:17, subclass).
  • ✅ FR-009 env overrides — RESOLVED divergence (trusted; spec updated 2026-07). llm/config.py docstring states "single source of truth is spike-profiles.yaml. No env variable overrides"; profile selected by SPIKE_PROFILE; PROVIDER_MODELS (config.py:38-65); REASONING_AGENTS = {"U:Ar","U:SD"} (:33).
  • ❌ FR-011 resolve_provider()/resolve_model() — ABSENT. symbols resolve_provider → 0; resolve_model only hit is providers/registry.py:80 (unrelated). Replaced by get_volk_provider (llm/config.py:323) / get_volk_model (:342).
  • ⚠️ Signature/return DIVERGENCE. Spec call_llm(volk_alias, messages, knowledge_graph_context, provider_override, model_override, reasoning_model_override, max_retries_override, metadata) -> LLMResponse; actual call_llm(system_prompt, user_message, model, temperature, volk, graph_context, env) -> str (:819). No LLMResponse/LLMRequest objects, no provider_override/model_override/reasoning_model_override/ max_retries_override/metadata params, no per-Volk ORCHESTRATA_LLM_MODEL_U:AR. The str return matches the spec's own US-028-001 ("Return type is a plain str") — the spec is internally inconsistent. FR-014 compliance_verified metadata is therefore not expressible (return is a bare str + warning log).
  • ⚠️ Duplicate config (orphaned parallel path). core/infra_config.py still honors ORCHESTRATA_LLM_PROVIDER/ _MODEL/_REASONING_MODEL (:114/:118/:120) and exposes get_volk_provider (:149)/get_volk_model (:156) with _REASONING_VOLKS = {"U:Ar"} (:101) — inconsistent with llm/config.py's REASONING_AGENTS={"U:Ar","U:SD"}. The LLM client + driver factory import orchestrata_ultimate.llm.config, not infra_config, so the env-override path is orphaned for LLM routing.

• Remaining work / plan: (spec update + a duplication cleanup decision — no functional hole)

  • Scope: rewrite SPEC-028 to the profiles-based redesign (SPIKE_PROFILE-selected spike-profiles.yaml, no env overrides, str-returning call_llm(system_prompt,user_message,…), get_volk_provider/get_volk_model in place of resolve_provider/resolve_model, MAX_RETRIES=6). Separately decide the fate of core/infra_config.py's duplicate LLM config (retire or wire as the deliberate CLI/test override layer).
  • Target files: specs/028-llm-client/spec.md; decision on core/infra_config.py LLM block. No change to llm/client.py/llm/config.py (intended design).
  • Approach: reconcile the spec text; for the infra_config duplicate, make a consolidation call (touches many env-config sites; relates to the "read env at call time" note).
  • Dependencies: SPEC-011 (ProviderManager), SPEC-025 (driver_factory imports llm.config), core/infra_config.
  • Effort: S (spec) / M (if infra_config consolidation is taken). Risk: low/med.

• Evidence log:

  • symbols call_llm|call_reasoner|resolve_provider|resolve_model → call_llm :819, call_reasoner :984, resolve_* absent (only registry.py:80).
  • source llm.client (head + retry + call_llm): MAX_RETRIES=6, _TRANSIENT_MARKERS, KG injection :852-857, compliance loop :892-960.
  • source llm.config → profile-based, PROVIDER_MODELS, REASONING_AGENTS, get_volk_provider:323/get_volk_model:342.
  • source core.infra_config → ORCHESTRATA_LLM_* env overrides :114-125, _REASONING_VOLKS={"U:Ar"} :101, get_volk_provider:149/get_volk_model:156.
  • symbols LLMClientError|LLMProviderUnavailableError → exceptions.py:8/:17.
  • refs call_llm → 11 production USE.

SPEC-053 OpenRouter Provider — DIVERGENCE (realized as an adapter under SPEC-011/028)

• Specifies: a standalone OpenRouterProvider (chat_completion/get_available_models/get_model_cost) owning multi-provider access, FALLBACK_CHAINS, RETRY_CONFIG, MODEL_PRICING/calculate_cost, budget enforcement/alerts, caching, audit — a monolithic provider subsystem.

• Verified:

  • ❌ No standalone OpenRouterProvider. symbols OpenRouterProvider → 0. No chat_completion / get_available_models / get_model_cost / call_with_fallback / MaxRetriesExceededError.
  • ❌ No spec constants. symbols FALLBACK_CHAINS → 0; MODEL_PRICING → 0; RETRY_CONFIG → 0. The only calculate_cost is SPEC-011's CostTracker.calculate_cost (cost_tracker.py:95), not a standalone function.
  • ✅ OpenRouter realized as an adapter. OpenRouterAdapter(BaseAdapter) (providers/adapters/openrouter_adapter.py:29) with complete (OpenAI-compatible POST /chat/completions at OPENROUTER_BASE_URL https://openrouter.ai/api/v1, HTTP-Referer/X-Title headers :37-48, tool-call forwarding :83-99), get_models (:127), supports_capability (:145). Wired — refs OpenRouterAdapter → llm/client.py:267 (import) + :312 (registered in _register_openrouter); also scripts/ask_openrouter.py.
  • 🔻 Responsibilities decomposed. Fallback chain + circuit breaker + cost tracking + budget → SPEC-011 (router.py, circuit_breaker.py, cost_tracker.py); retry/backoff → SPEC-028 (_execute_with_retry). FR-002/003/004/005/007 exist system-wide, not OpenRouter-owned. Per-agent routing = spike-profiles.yaml; OpenRouter-specific escalation = OPENROUTER_MODEL_ESCALATIONS (llm/config.py:70-79). PROVIDER_MODELS["openrouter"] (config.py:48-51) registers qwen family models.
  • ❌ FR-011 caching absent. No model-list (1h) / pricing (24h) / routing (1h) cache in openrouter_adapter.py or registry.py.
  • 🔻 External-integration points out of scope for this codebase. REQ-053-052 REST API (GET /api/v1/models etc.), REQ-053-071/072 (Postgres/RabbitMQ/Stripe/Kong/ELK) are integration contracts from a pre-port spec with no src/ realization and no consumer.

• Remaining work / plan: (spec reconciliation — no real code work)

  • Scope: rewrite/retire SPEC-053 to state "OpenRouter is an IProvider adapter (OpenRouterAdapter) plugged into the SPEC-011 ProviderManager; routing/fallback/cost/budget/retry are owned by SPEC-011 + SPEC-028." Drop the OpenRouterProvider class contract, FALLBACK_CHAINS/MODEL_PRICING/RETRY_CONFIG duplication, and the Postgres/RabbitMQ/Stripe/Kong integration points. Optionally implement model-list/pricing caching if the perf target is wanted (that would be S/M code, not spec).
  • Target files: specs/053-openrouter-provider/spec.md (mark superseded-by-011+028); optional openrouter_adapter.py/registry.py cache.
  • Approach: reconcile SPEC-011/028/053 in one pass (053 overlaps both).
  • Dependencies: SPEC-011 + SPEC-028.
  • Effort: S (spec) / S–M (if caching added). Risk: low.

• Evidence log:

  • symbols OpenRouterProvider → 0; symbols FALLBACK_CHAINS/MODEL_PRICING/RETRY_CONFIG → 0.
  • symbols OpenRouterAdapter → openrouter_adapter.py:29; refs OpenRouterAdapter → llm/client.py:267/:312.
  • source providers.adapters.openrouter_adapter → OpenAI-compatible endpoint, headers, get_models, supports_capability.
  • source llm.config → OPENROUTER_MODEL_ESCALATIONS, PROVIDER_MODELS["openrouter"].

Cluster summary table

Notes / non-gaps (do NOT "fix")

• memory/token_counter.py:59 except ImportError, KeyError: and deepseek_coder_driver.py:505 except OSError, UnicodeDecodeError: remain valid PEP-758 parenthesis-free except on Python 3.14 — not bugs.
• _execute_with_retry now carries a cassette layer (LLM_CASSETTE_DIR/LLM_CASSETTE_SYNTH/LLM_CASSETTE_RECORD) and a 900s total retry budget — the 2026-08-05 CXP accept-queue wedge mitigation. Not spec-required; not a gap.

Cross-cluster dependencies noticed

• SPEC-011 ↔ SPEC-010: both need the same unresolved session.closed → EventBus/persistent-engine wiring (SPEC-011 Neo4j cost cold-migration; SPEC-010 FR-008 memory migrate_to_cold). Solve once.
• SPEC-011 ↔ governance (SPEC-013 PAM++/janus): FR-013 model-access + budget-override predicates depend on the Prolog-janus adapter.
• SPEC-011 ↔ SPEC-001 EventBus: provider event emission depends on kernel.emit_event.
• SPEC-028 ↔ core/infra_config: a duplicate env-honoring get_volk_provider/get_volk_model coexists with the live llm/config.py; consolidation is a design decision. Touches SPEC-025 (driver_factory imports llm.config).
• SPEC-053 ⊂ SPEC-011 + SPEC-028: reconcile the three specs in one pass.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks