Orchestrata Ultimate › Plans › port-from-orchestrata

Implementation Plan: Port Orchestrata to Orchestrata-Ultimate

Source: plans/port-from-orchestrata.md at 8a2cb42ad · last changed 2026-08-10 · first committed 2026-04-11 · 514 lines.

Prose pending.

Plan text

Implementation Plan: Port Orchestrata to Orchestrata-Ultimate

Executive Summary

Port ~50,000 LOC of tested implementations from orchestrata's 12 packages into orchestrata-ultimate's package structure. Keep ultimate's core/ infrastructure (observability, error handling, threading, exceptions). Replace all stubs with real implementations. Adapt imports, interfaces, and external dependencies.

Source: /home/mohamed/workspace/orchestrata/packages/orchestrata-*/src/orchestrata/ Target: /home/mohamed/workspace/orchestrata-ultimate/src/orchestrata_ultimate/


Current State

What Ultimate Has (KEEP)

Package LOC Status Action
core/logging/ 1,063 Real (structlog, audit, context) KEEP
core/metrics/ 780 Real (Prometheus, decorators) KEEP
core/tracing/ 583 Real (OpenTelemetry) KEEP
core/patterns/error_handling.py 729 Real (retry, circuit breaker, bulkhead) KEEP + fix bugs
core/threading/ 892 Real (thread pool, task tracking) KEEP + fix bugs
core/domain/ 214 Real (Node, DomainEntity) KEEP
exceptions/ 154 Real (exception hierarchy) KEEP + extend

What Ultimate Has (REPLACE WITH ORCHESTRATA)

Package LOC Status Action
agents/ 538 100% stub REPLACE
integrations/cheetahclaws.py 136 100% stub REPLACE
integrations/openrouter.py 80 100% stub REPLACE
integrations/todoist.py 75 100% stub REPLACE
integrations/vizint.py 76 100% stub REPLACE
knowledge_graph/graph.py 110 100% stub REPLACE
governance/pam.py 145 Shallow REPLACE
core/orchestrator.py 279 Framework only REPLACE
cli.py 290 Display only REPLACE

What Orchestrata Has (PORT)

Package LOC Files Key Modules
orchestrata-kernel 15,272 24 EventBus, SessionManager, Pipeline, Workflow, AuditLogger, PAMEnforcer, VolkRegistry
orchestrata-cxp 9,525 34 CXPPacket, JSONSerializer, Validator, ThreadManager, PostmanAdapter
orchestrata-knowledge 8,449 35 Neo4jClient, CodeParser, Synchronizer, ContextInjector, AnomalyDetector
orchestrata-governance 5,736 19 PolicyEngine, EscalationHandler, ConstitutionalEnforcer, PrologClient
orchestrata-agents 5,353 31 BaseVolk, VolkRuntime, VolkPool, TaskExecutor, Coordinator, Hierarchy
orchestrata-gateway 4,429 24 WebSocketGateway, RPCHandler, PubSub, Auth, Handlers
orchestrata-plugin 3,599 13 PluginLoader, PluginManager, Registry, SDK
orchestrata-drivers 2,070 10 ClaudeCodeDriver, AiderDriver, DeepSeekDriver, BrowserDriver
orchestrata-provider 1,936 14 ProviderManager, Adapters (6), CostTracker, RateLimiter
orchestrata-postman 12,674 31 Postman, Router, DeliveryManager, QueueManager, Guarantees
orchestrata-session 10,595 37 Session lifecycle, WorkItem, Goals, Checkpoint, KG context
orchestrata-memory 1,145 6 MemoryEngine, HotStore, ColdStore, Compressor
orchestrata-llm 722 4 call_llm(), call_reasoner(), VolkConfig
orchestrata-compliance 766 3 ComplianceVolk, Pipeline
orchestrata-concierge 381 2 ConciergeVolk
TOTAL ~83,000 ~287

Module Mapping Table

Package-Level Mapping

Orchestrata Source Ultimate Target Notes
orchestrata.cxp.* orchestrata_ultimate.cxp.* New package — core protocol
orchestrata.postman.* orchestrata_ultimate.postman.* New package — message broker
orchestrata.kernel.* orchestrata_ultimate.kernel.* New package — orchestration kernel
orchestrata.agents.* orchestrata_ultimate.agents.* Replace stubs
orchestrata.governance.* orchestrata_ultimate.governance.* Replace shallow pam.py
orchestrata.knowledge.* orchestrata_ultimate.knowledge_graph.* Replace stub graph.py
orchestrata.session.* orchestrata_ultimate.session.* New package
orchestrata.memory.* orchestrata_ultimate.memory.* New package
orchestrata.provider.* orchestrata_ultimate.providers.* Replace stub openrouter.py
orchestrata.llm.* orchestrata_ultimate.llm.* New package
orchestrata.gateway.* orchestrata_ultimate.gateway.* New package
orchestrata.plugin.* orchestrata_ultimate.plugin.* New package
orchestrata.drivers.* orchestrata_ultimate.drivers.* New package
orchestrata.volks.compliance.* orchestrata_ultimate.agents.compliance.* Under agents/
orchestrata.volks.concierge.* orchestrata_ultimate.agents.concierge.* Under agents/

Class-Level Mapping

Orchestrata Class Ultimate Class Change Required
BaseVolk(BaseEntity) BaseAgent(BaseModel) Unify: keep Pydantic, add CXP send/receive
AnalystVolk AnalystAgent Rename, adapt imports
SeniorDevVolk SeniorDevAgent Rename, adapt imports
JuniorVolk JuniorAgent Rename, adapt imports
CodeDirectorVolk CodeDirectorAgent Replace current stub
QaVolk QualityAssuranceAgent Replace current stub
ArchitectVolk ArchitectAgent Replace current stub
UXVolk UserExperienceAgent Replace current stub
PMVolk ProjectManagerAgent New in ultimate
ConciergeVolk ConciergeAgent Under agents/
ComplianceVolk ComplianceAgent Under agents/
CXPPacket CXPPacket Direct port
Postman(IPostman) Postman Remove interface, keep impl
OrchestrationKernel OrchestrataKernel Merge with existing
PolicyEngine GovernanceEngine Replace current shallow impl
AdvancedEscalationHandler EscalationHandler New in ultimate
Neo4jClient Neo4jGraph Replace current stub
ProviderManager ProviderManager Replace current OpenRouter stub
MemoryEngine MemoryEngine New in ultimate
SessionManager SessionManager New in ultimate
EventBus EventBus New in ultimate
VolkRuntime AgentRuntime Rename
VolkPool AgentPool Rename
VolkRegistry AgentRegistry Rename
TaskExecutor TaskExecutor Direct port
VolkCoordinator AgentCoordinator Rename
PluginManager PluginManager Direct port
WebSocketGateway WebSocketGateway Adapt to FastAPI

Interface Removal

Orchestrata uses abstract interfaces (IPostman, IOrchestrator, IPacket, etc.) defined in orchestrata.core.interfaces. Ultimate uses concrete classes directly.

Orchestrata Interface Action
IOrchestrator Remove — use OrchestrataKernel directly
IPostman Remove — use Postman directly
IPacket Remove — use CXPPacket directly
ISerializer Remove — use JSONSerializer directly
IValidator Remove — use CXPValidator directly
IEventBus Remove — use EventBus directly
IPlugin Keep — plugins need interface contract
IProvider Remove — use concrete adapters
IPolicyEngine Remove — use GovernanceEngine directly

Import Translation Rules

from orchestrata.core.interfaces import *     → DELETE (no interfaces)
from orchestrata.core.traceability import *   → from orchestrata_ultimate.core.tracing import *
from orchestrata.core.infra_config import *   → from orchestrata_ultimate.core.config import *
from orchestrata.cxp.packet import *          → from orchestrata_ultimate.cxp.packet import *
from orchestrata.postman.* import *           → from orchestrata_ultimate.postman.* import *
from orchestrata.kernel.* import *            → from orchestrata_ultimate.kernel.* import *
from orchestrata.agents.* import *            → from orchestrata_ultimate.agents.* import *
from orchestrata.governance.* import *        → from orchestrata_ultimate.governance.* import *
from orchestrata.knowledge.* import *         → from orchestrata_ultimate.knowledge_graph.* import *
from orchestrata.session.* import *           → from orchestrata_ultimate.session.* import *
from orchestrata.memory.* import *            → from orchestrata_ultimate.memory.* import *
from orchestrata.provider.* import *          → from orchestrata_ultimate.providers.* import *
from orchestrata.llm.* import *              → from orchestrata_ultimate.llm.* import *
from orchestrata.gateway.* import *           → from orchestrata_ultimate.gateway.* import *
from orchestrata.plugin.* import *            → from orchestrata_ultimate.plugin.* import *
from orchestrata.drivers.* import *           → from orchestrata_ultimate.drivers.* import *

Phased Implementation

Phase 1: Foundation (CXP + LLM + Provider)

Goal: Get agents talking to each other and to LLMs. Estimated LOC: ~12,000

Step 1.1: Port CXP Protocol
Step 1.2: Port Provider Manager + LLM Client
Step 1.3: Port Agent System

Phase 2: Infrastructure (Postman + Kernel + Session)

Goal: Message routing, session management, event-driven coordination. Estimated LOC: ~38,000

Step 2.1: Port Postman Message Broker
Step 2.2: Port Kernel
Step 2.3: Port Session Package

Phase 3: Intelligence (Governance + Knowledge + Memory)

Goal: Policy enforcement, code understanding, persistent memory. Estimated LOC: ~15,000

Step 3.1: Port Governance
Step 3.2: Port Knowledge Graph
Step 3.3: Port Memory Engine

Phase 4: Interface (Gateway + Plugin + Drivers + Integrations)

Goal: WebSocket API, plugin system, code generation, external services. Estimated LOC: ~10,000

Step 4.1: Port Gateway
Step 4.2: Port Plugin SDK
Step 4.3: Port Code Generation Drivers
Step 4.4: Build Real Integrations

Dependency Cleanup

Remove Unused Dependencies (Current)

# These are listed but nothing uses them yet:
# grpcio>=1.80.0        — Skip until gRPC transport needed
# grpcio-tools>=1.80.0  — Skip until gRPC transport needed  
# asyncio>=3.4.3        — stdlib, PyPI package is no-op
# msgpack>=1.0          — Skip until binary serialization needed

Add When Needed (Phase-Dependent)

# Phase 1: Already in pyproject.toml
# anthropic, openai, httpx, pydantic — already listed

# Phase 2: Already listed
# redis, websockets, fastapi, uvicorn — already listed

# Phase 3: Already listed  
# neo4j, pyswip — already listed

# No new dependencies needed — orchestrata's deps are a subset of ultimate's

Bug Fixes (Pre-Port)

Fix these in ultimate's core/ before porting:

Bug File Fix
Wrong import path core/logging/logging_config.py:71 core.context → core.logging.context
Deprecated API (6 locations) core/patterns/error_handling.py asyncio.iscoroutinefunction → inspect.iscoroutinefunction
Race condition error_handling.py:307,348 Await asyncio.create_task() in circuit breaker
Race condition thread_pool.py:430,441 Protect _active_task_ids with lock
Memory leak thread_pool.py Add _tasks cleanup (completed tasks > 1000)
Pydantic v1 pattern domain/node.py:40 Config → model_config = ConfigDict(...)
Pydantic v1 method domain/domain_entity.py .dict() → .model_dump()
Duration not recorded metrics/decorators.py:144-149 Actually record duration in metric
No-op cleanup metrics_config.py:304-307 Implement _stop_system_metrics_collection()
Hardcoded after_state audit.py:376-379 Capture real state in AuditOperation.__exit__

Test Strategy

Port Tests Alongside Code

Each phase ports tests from orchestrata alongside the implementation:

Phase Orchestrata Test Files Test Functions
1 (CXP + LLM + Agents) ~100 ~3,500
2 (Postman + Kernel + Session) ~160 ~5,000
3 (Governance + Knowledge + Memory) ~100 ~2,000
4 (Gateway + Plugin + Drivers) ~50 ~1,400
Total ~410 ~11,900

Import Translation for Tests

from orchestrata.cxp import *     → from orchestrata_ultimate.cxp import *
from orchestrata.agents import *  → from orchestrata_ultimate.agents import *
# ... same pattern as source

Test Validation

After each step, run:

uv run pytest tests/ -v --tb=short

All existing passing tests must continue to pass. New ported tests must pass.


Duplicate Spec Resolution

Pair Keep Archive Reason
000-orchestrata-cli / 000-ultimate-cli 000-ultimate-cli 000-orchestrata-cli Ultimate CLI is the target
003-intent / 003-intent-broker 003-intent-broker 003-intent Broker spec is more detailed
007-prolog-governance / 035-verification Both Neither Different scopes (governance vs verification agent)
001-kernel / 001-ultimate-architecture 001-ultimate-architecture 001-kernel Architecture spec subsumes kernel

File Deletion Checklist

After each phase, delete the stubs that were replaced:

After Phase 1

After Phase 2

After Phase 3

After Phase 4


Success Criteria

Metric Before After
Source LOC 1,985 ~55,000
Real implementations 7 modules 15+ packages
Stub modules 9 0
Test functions 213 passing ~12,000 passing
Unused dependencies 8 0
Agents with LLM calls 0 9+
Neo4j integration Stub Real (Cypher, AST sync)
Governance Substring matching PolicyEngine + Prolog + Escalation
Message broker None Postman with routing + guarantees
CLI Display-only Functional commands

Risks

Risk Mitigation
grpcio won't build on Python 3.14t Skip gRPC transport, use JSON over HTTP/WebSocket
Prolog (pyswip) availability Make Prolog optional with fallback to Python rule engine
Redis not available locally Use in-memory implementations as default, Redis as optional
Neo4j not available locally Use in-memory graph for dev, Neo4j for integration tests
Import volume (287 files to translate) Script the bulk sed replacement, fix manually where needed
Circular dependency during port Port in dependency order: CXP → Provider → Agents → Postman → Kernel

Plan created: 2026-04-10 Source analysis: 12 packages, ~83,000 LOC, 287 files, 11,914 tests