Orchestrata Ultimate › Plans › implementation-status-and-plan

Implementation Status & Plan — orchestrata-ultimate

Source: plans/implementation-status-and-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-06-20 · 213 lines.

Prose pending.

Plan text

Implementation Status & Plan — orchestrata-ultimate

Produced from three read-only audits (volk completeness vs. frozen source; non-volk stub inventory; intended-set + wiring gaps), each spot-verified against the code. Date: 2026-06-20.

TL;DR

STATUS UPDATE (2026-06-20, Session 165): the wiring described below is now BUILT. Phases A/B/C are complete and Phase D is all-but-one done (only SPEC-017 Calendar remains). The "gateway is wired to nothing" framing below is the original diagnosis, kept for history — see Part 3 §C and the Phase A–D sections for current per-item status. Trust the code (and CHANGELOG Session 165) over any percentage in this doc.

The premise "the volks are missing except architect" is false at the code level but true in spirit: all 9 volks are fully implemented, but none of them actually run from the UI because the gateway is wired to nothing. The gap is integration/wiring, not missing volks. Individually every engine component exists; they were never connected into a working request path. That is why the system looks unimplemented. (Original diagnosis — resolved by Phase A.)

Decisions locked (2026-06-20)

• Topology = out-of-process gRPC. Honor the docker-compose service split: the gateway (orchestrata-api) connects over gRPC to a separate agent-runtime CXP kernel and a separate knowledge-graph service. Phase A wires the gateway with a GRPCClient and gives the kernel's CXPServer a real packet_handler that runs the volks — not an in-process PipelineRunner.
• Containers = Incus, not Docker. docker-compose.yml is the topology reference only; deployment + harness isolation target Incus. Implementing harness/isolation.py IncusIsolation (today NotImplementedError) is in-scope.
• Ports = single source in .env, no hardcoded defaults — DONE (commit 8366b185). CXP_GRPC_PORT / KG_GRPC_PORT live in .env; code reads them at call time via cxp/endpoint_config.py and raises CXPConfigError if unset. This supersedes the earlier "pin everything to 50051" fix and the W5/W7 port notes below.

Part 1 — What IS implemented (verified)

The 9 volks — all COMPLETE

Real LLM/driver calls, real I/O, full method parity with the frozen per-volk source packages (orchestrata-architect, orchestrata-qa, … — not the legacy orchestrata-agents/volks/ dir, which only ever had 4).

raise NotImplementedError, canned returns, and pass-only public methods: none in any volk. The """Volk stub: …""" docstrings in 5 __init__.py files are byte-identical to source and harmless (the classes are imported from volk.py).

Engine components — all individually implemented

PipelineRunner (kernel/pipeline.py, instantiates all 9 volks on a Postman, routes CXP packets via submit_intent), Postman, CXPServer/GRPCClient (recently fixed), MessageDispatcher, VolkRegistry, VolkRuntime/VolkPool, TaskExecutor, IntentBroker, PolicyEngine/EventEmittingPolicyEngine, governance Prolog gate, knowledge_graph, session, memory. All real. The problem is they're not connected.

Part 2 — The real gap: integration / wiring

There are three independent "run a volk" paths and the gateway (UI front door) is wired to none of them. Verified:

Why architect seemed "the one that works": nothing in the UI path runs any volk; any prior signs of life came from a direct/CLI invocation. The volks aren't the problem — the wiring is.

Part 3 — Complete inventory of unimplemented parts

A. Volk-level (1 real regression + cosmetic nits)

• Concierge answer-replay dedup dropped — source _seen_answers/prior_answers (volk.py:136,228) not ported (port volk.py:128,227). Real, small.
• Nits: architect architect_context.py:295 hardcodes /opt/orchestrata/packages; SeniorDevAgent.constraints = [] class-level mutable default (senior_dev.py:51); orphaned test-only mixins (analyst_parsing/storage); dead-but-faithful JS checks in qa/ux.

B. Subsystem stubs (~12 real gaps)

Intentional (leave as-is, documented): harness/isolation.py:127 IncusIsolation, abstract bases (agents/cxp_adapter.py:86, kernel/audit/writers.py:38, harness/models.py:189), CLI status stubs (kernel/cli.py:94, postman/cli.py:102, kernel/console_cli.py), generated proto, import fallbacks, di generic simplification.

C. Specced feature status (CORRECTED 2026-06-20 — verified against code, not RTM)

The "0%" figures previously here were WRONG: SPEC-103 and SPEC-104 were fully built in Session 155 but this doc still claimed no src. Verified each against the actual code on 2026-06-20.

• SPEC-103 Tool-Calling Refactor — ✅ DONE (Session 155). src/orchestrata_ultimate/refactor/ (agent.py OBSERVE→MUTATE→VERIFY loop, tools.py 7 sandboxed tools + dispatch), DeepSeek tool-calling in the provider layer, REFACTOR intent wired in intent_broker.py + classifier precedence; scripts/refactor.py CLI; tests/refactor/test_tools.py (21 pass).
• SPEC-104 OpenSpec Format — ✅ DONE (Session 155, commit 3cce7854). src/orchestrata_ultimate/openspec/ (detect/parse/render/repo); detect, parse, render, validate, delta-edit with sandboxed I/O; tests/openspec/test_openspec.py (16 pass). (The earlier "openspec/ dir is unrelated" note was false.)
• SPEC-018 Fireflies — ✅ DONE (Session 165). Read-side handler + write-side KG sync (integrations/fireflies_sync.py, idempotent :Document MERGE); real-Neo4j tests.
• SPEC-015 Plugin SDK — ✅ DONE (Session 165). Runtime load/registration now wires into real registries (PluginManager accepts them; plugin/registries.py::VolkRegistry); tested end-to-end. (pip/git install still NotImplementedError — local works; tracked in Part B Low.)
• SPEC-010 Memory Engine — ✅ DONE (Session 165). Engine was complete; now reachable via memory.* gateway RPC (gateway/handlers/memory.py); real-Redis tests. (Kernel/volk-pipeline use of the engine is a follow-up.)
• SPEC-017 Calendar Integration — ✅ DONE (Session 165). Browser-side gadget vizint-ui/gadgets/calendar/ (ical parse/build, day-view timeline, real CalDAVProvider + GoogleCalendarProvider, SessionCalendarSync); registered in app.js; index.html CSP extended for calendar origins. Real-infra tested: 21 vitest units + 3 Playwright e2e against a real local CalDAV server (tests/e2e/caldav_server.py), no fetch mocking. Follow-ups: Google OAuth e2e (needs real Google), remote-HTTPS-CalDAV CSP/proxy.

Part 4 — Implementation plan (phased)

Phase A — Wire the engine into the gateway (highest impact; makes volks actually run) — ✅ COMPLETE (Session 165)

Goal: a UI intent reaches a live volk and a real reply returns.

1. Engine path = out-of-process gRPC (per Decisions locked). The agent-runtime process runs the CXP kernel; give its CXPServer a real packet_handler that drives the volks (e.g. via PipelineRunner, which already builds all 9 volks on a Postman with submit_intent). The gateway does NOT instantiate the engine in-process.
2. Gateway → kernel client. In gateway/server.py construct a GRPCClient / MessageDispatcher to cxp_grpc_endpoint() (the agent-runtime kernel) and inject it into the handler context. (Ports already resolve from .env — commit 8366b185.)
3. W2: rewrite orchestrator.handle_intent to call engine.submit_intent(...) and return the real task/stream id (keep the Neo4j card as a side-record, not the result).
4. W3: replace cxp._maybe_reply_as_concierge canned reply with a real ConciergeVolk.process() call routed through the engine; stream the CXP reply packet.
5. W7: reconcile volk aliases to ONE canonical set (adopt pipeline's J:Junior/U:Cp everywhere; fix task_executor._VOLK_CLASS_MAP). Add a test asserting the executor map keys == pipeline registration keys == gateway default-registry aliases.
6. W5: pass a real packet_handler to CXPServer(config, packet_handler=…) in runtime_main.py that dispatches into the engine; add an integration test that a packet sent via GRPCClient is actually processed (not dropped).
7. W6: in cli_main._boot_monolithic, register all volks into the VolkRegistry and actually use the TaskExecutor/IntentBroker (remove the # noqa: F841 discards), or delete the dead path if Phase A standardizes on PipelineRunner.

• Acceptance: end-to-end test — UI orchestrator.intent (or cxp.send) → Concierge runs → reply returns; plus the alias-consistency test. (Reuse the /run debug-stack demo from this session to validate against live Redis/Neo4j.)

Phase B — Close the real subsystem stubs — ✅ COMPLETE (Session 165; Low-priority cluster left as documented harmless dead code)

1. Governance (W4): instantiate PolicyEngine once; wire handle_evaluate to a real dry-run and handle_rules to the engine's active rule set.
2. claude/hooks.py:122: resolve config hooks to real callables (import path / registry) instead of a no-op lambda; error loudly on unresolvable hooks.
3. postman/delivery.py:457: persist the original payload so retry re-delivers it.
4. cxp.handle_status: back it with real delivery/event tracking from the dispatcher.
5. volk.register/deregister: wire to VolkRegistry.register_volk/deregister_volk behind an admin-scope check.
6. Lower priority: _query_prolog (delete or implement), filter headers, escalation vote/priority strategies, test_runner coverage.py, plugin pip/git install.

• Acceptance: a regression test per gap (pattern: the test_grpc_kernel_endpoint.py guard added this session — reintroduce-the-bug-and-watch-it-fail).

Phase C — Correctness nits — ✅ COMPLETE (Session 165; the SeniorDev "mutable default" was a verified Pydantic false-positive)

• Restore Concierge _seen_answers/prior_answers dedup (+ test for replayed answers).
• Fix SeniorDevAgent.constraints mutable default (→ None + per-instance init).
• Resolve the architect /opt/orchestrata/packages hardcoded path (env/config).
• Remove orphaned test-only mixins or fold their tests onto the live classes.

Phase D — Specced features — ✅ COMPLETE (2026-06-20)

SPEC-103 + SPEC-104 were already complete (Session 155 — this plan's "greenfield" label was wrong). SPEC-018 / SPEC-015 / SPEC-010 and the calendar gadget SPEC-017 were completed in Session 165 (the calendar via a real-CalDAV Playwright E2E). Remaining follow-ups are narrow and documented per spec above (Google-OAuth e2e, remote-HTTPS-CalDAV CSP/proxy, projecting plugin/memory registries onto the live kernel dispatcher).

Suggested order

A (wiring) → B (stubs) → C (nits) → D (specs). Phase A is the unlock: it converts a demo facade into a working product and is what makes the volks — already built — actually do their jobs. Everything in A/B/C is verified-real work on existing code; D is new scope.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks