Orchestrata Ultimate › Plans › architecture-review-2026-07-18

Orchestrata Ultimate — Architecture Review & Gap-Closure Design

Source: plans/architecture-review-2026-07-18.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-18 · 202 lines.

Prose pending.

Plan text

Orchestrata Ultimate — Architecture Review & Gap-Closure Design

Method: the system-design skill applied to the actual system (Phase 0 = ground truth first). Evidence gathered with the repo's own AST tooling (plans/ast-ultimate.json, direct query) for structure + grep for call-sites (the AST dependency graph is unreliable for instantiation, per kernel-consolidation-plan.md). Graded against live code @ master 3b15967 (2026-07-18), not spec checklists / @traces / the auto-audit (all unmaintained). Companion: ORCHESTRATA_UNDERSTANDING.md. Governing constraint (Mohamed): unimplemented FRs are implementation GAPS to be WIRED — never removed. This document proposes zero deletions of spec-mandated capability. Where a duplicate implementation exists, the recommendation is wire-or-quarantine-to-experimental/ (the repo's own D1 pattern), never delete.

0. Ground truth (Phase 0)

Spec-driven multi-agent SWE runtime; single-package modular monolith orchestrata_ultimate; Python 3.14t (free-threaded, GIL off); uv. AST snapshot: 695 modules · 1,322 classes · 5,252 methods · 180,604 LOC. Six layers (0 Vocabulary → 5 Interface), "down only, sideways through protocols only." The kernel is the never-retired Layer-4 runtime the volks execute on; PipelineRunner (kernel/pipeline.py:96) is the canonical engine. Fixed deterministic pipeline; all inter-agent comms via CXP→Postman→Redis; governance via PAM++ Prolog (PrologGate, janus). Three-tier durable memory: Neo4j (operational) + Memgraph-HA orchgraph (semantic/RAG) + Wiki.js. Full validated model in the companion doc §1–§19.

Diagram source shown as text: Mermaid could not parse it (Error: Parse error on line 34:).

flowchart TB
  H([Human intent]) --> CX
  subgraph L5[Layer 5 · Interface]
    CLI[orchestrata CLI]:::i
    GW[WebSocket Gateway<br/>+ DI container · IMemoryEngine singleton]:::i
    VZ[VizInt UI]:::i
  end
  subgraph L4[Layer 4 · Orchestration]
    PR[PipelineRunner<br/>kernel/pipeline.py:96 · canonical engine]:::k
    ORCH[orchestrator/pipeline_runner.py<br/>+ PipelineVerificationGate enforce=False]:::k
    SESS[Session + Memory<br/>checkpoint/recovery/migrate_to_cold]:::k
    KRN[[KERNEL runtime · boot/registry/gRPC :50053/PrologGate/audit]]:::heart
  end
  subgraph L3[Layer 3 · Domain Services]
    VOLKS[Volks: Cx→BA→PM→Ar→Cd→Qa→Cp<br/>+ Ux cond. · never import each other]:::s
    GOV[Governance · PrologGate/PAM++ · janus]:::s
    KG[Knowledge · Neo4j + Memgraph-HA + Wiki.js]:::s
    PROV[Providers · Circuit Breaker + fallback]:::s
    VER[Verification · VerificationCoordinator]:::s
  end
  subgraph L2[Layer 2 · Communication]
    CXP[CXP packets]:::c --> POST[Postman] --> RDS[(Redis)]
  end
  CX[U:Cx Concierge] --> VOLKS
  CLI --> PR
  GW --> PR
  PR --> KRN
  PR -.governs each stage.-> GOV
  VOLKS -->|CXP only| CXP
  ORCH -.verdicts computed, not enforced.-> VER
  SESS --> KG
  VOLKS --> PROV
  classDef heart fill:#b91c1c,color:#fff,stroke:#7f1d1d,stroke-width:3px;
  classDef k fill:#1e3a8a,color:#fff; classDef s fill:#065f46,color:#fff;
  classDef c fill:#92400e,color:#fff; classDef i fill:#374151,color:#fff;

1. Headline finding — the gap list is a MOVING TARGET; the wire-not-retire program is working

The most important review result: several capabilities flagged "orphan / 0-use" by the spec-audit and the spec-synthesis have since been WIRED. Confirmed by grep against current src/:

Implication for the design: do not drive gap-closure from the audit/synthesis lists — they lag the code. Re-verify each candidate against live source (as below) before scheduling work. This is precisely why AST+grep review beats reading the planning artifacts: the artifacts describe intent at a past session; the code is now.

2. Genuinely-open gaps (verified current) — each a WIRE, none a retire

Ordered by leverage. Each: the gap (evidence), the wiring design, the named pattern, the seam (file:line), the trade-off, and alignment with the repo's own swarm/keystone plan.

G1 — SPEC-072 ToolSearchTool is registered/dispatchable but not exposed to any volk ✅ WIRED in this change

• Corrected evidence (read the code, not just grep): tools/search.py:1467 defines it and it is already registered + dispatchable — register_native_tools builds it via the .create() factory and registers it under the name "search_tools" (agents/tool_registry.py:1979; tools/search.py:1570). My initial grep 'ToolSearchTool(' reported "0 sites" because it missed the classmethod-factory call ToolSearchTool.create(registry) — a good reminder that grep finds constructor calls, not factory wiring. The actual gap is exactly what the code's own docstring names: "Volk-facing exposure through AGENT_TOOLS is a separate, governance-gated decision" (:1929). No volk's AGENT_TOOLS list contained search_tools, so no volk could invoke it (get_agent_tools → :1833 allowed = AGENT_TOOLS.get(alias, [])).
• Wire (done): added "search_tools" to AGENT_TOOLS for U:BA (:544), U:Ar (:571), U:Cd (:592) — the analysis/design/implementation volks that select tools under uncertainty. Exposure resolves to a live schema through the real get_agent_tools → _native_schema_registry() → register_native_tools path (:2004), and to_function_schemas skips unknown names (tools/registry.py:104), so the change is safe by construction. Registration itself needed no change — it was already done.
• Tests (no mocks, real path): test_planning_volks_can_search_tools asserts the three volks now see search_tools; test_elicitation_volk_has_no_tool_search locks the scoping decision (U:Cx stays without it).
• Pattern: Service Discovery for tools (agent queries a registry at plan-time) — the same idea this runtime uses for MCP tool search. Trade-off: adds one optional discovery call on plan steps; scoped to the three volks that benefit, not U:Cx (elicitation doesn't need it). Requirement: SPEC-072 FR (agents discover tools dynamically vs a hardcoded allow-list). Aligns with swarm S1 Toolsmith. Effort: S — completed.

G2 — Verification gate is wired but non-enforcing (enforce=False)

• Evidence: orchestrator/pipeline_runner.py:223,232 constructs PipelineVerificationGate, :331,478 calls open_session; but verification_gate.py:25 — "default VerificationGateConfig.enforce=False — verdicts are computed" and not gated.
• This is not an orphan and not a defect — it's the correct rollout state. Shipping a new gate in shadow mode (compute verdicts, don't block) is the textbook safe path.
• Wire (the remaining step, a product decision for Mohamed): run shadow long enough to trust the verdicts (measure false-block rate on real dogfood intents), then flip enforce=True behind config — ideally per-stage (enforce on U:Cd/U:Qa outputs first). Pattern: Shadow → Canary → Enforce (progressive delivery of a policy gate). Trade-off: enforcing too early blocks valid builds on immature verdicts; too late means the gate adds latency for no protection. Recommend a burn-in metric (shadow_block_rate < X% over N intents) as the flip criterion — record as a decision (D-A below). Aligns with S10 Sentinel / audit D3. Effort: M.

G3 — Compliance gate is a default no-op

• Evidence: agents/volks/compliance/volk.py:65 self.compliance_threshold: float = 0.0; :200 passed = coverage_pct >= self.compliance_threshold → always True.
• Assessment: either an intentional soft-launch (consistent with G2's shadow philosophy) or drift. Not a wiring gap — the volk is in the pipeline (U:Cp terminal stage). It's a threshold-configuration decision.
• Recommend: make the threshold config-driven (env / governance fact) with a non-zero default once the compliance signal is trusted; until then, log the computed coverage vs a shadow threshold so the gate is observable while non-blocking (mirror G2). Trade-off / requirement: SPEC-026 implies a real gate; a 0.0 default silently satisfies the letter while voiding the intent — surface it explicitly rather than leave it implicit. Effort: S.

G4 — CheckpointManager not proven on the CheetahClaws edit/rollback UX path

• Evidence: wired for session recovery (session/recovery.py:30), but SPEC-050/061's P0 checkpoint-capture
  • <1s rollback for the /edit flow (files + conversation + KG) isn't confirmed on that path.
• Wire: reuse the existing CheckpointManager from the UltimateTUI edit handler — capture before a multi-agent edit commits, restore on rollback. Pattern: Memento / Snapshot + Compensating Transaction (the edit's parallel-review reject path already fails-closed; checkpoint gives it a clean undo). Trade-off: capturing conversation+KG per edit costs latency/storage; bound it (last-N checkpoints, async KG snapshot). Effort: M. Reuses built code — no new subsystem.

G5 — PAMEnforcer is a superseded duplicate, not a governance hole

• Evidence: kernel/pam/enforcer.py:49 defined, 0 instantiation; the live governance path is governance.prolog_gate.PrologGate (kernel/pipeline.py:31,310 self.prolog.may_execute(...)).
• Governance is enforced. PAMEnforcer is a second implementation of the same responsibility. This is the exact shape of audit D1 (0-ref duplicate shadowing the canonical path).
• Decision (not deletion): either (a) wire any capability PAMEnforcer has that PrologGate lacks into PrologGate, then (b) quarantine the emptied duplicate to experimental/ (preserved, excluded from check-deps) — never delete spec-mandated code outright. Pattern: Strangler Fig (the canonical PrologGate has already grown around the old enforcer; finish the strangle). Record as D-B. Effort: S.

3. Cross-cutting review — Well-Architected pillars (Phase 5 lens)

• Reliability: Providers already implement Circuit Breaker + fallback (good). The two-graph topology is deliberately kept with 5 gated prereqs before consolidation (sound — don't migrate split-brain onto a new engine). Watch: the dual PipelineRunner (kernel vs orchestrator) is an in-flight Strangler migration; until it lands, verification (G2) lives only on the orchestrator runner — confirm which runner the live CLI path uses so the gate isn't bypassed. Requirement: the escalation-absorption fork must sign off before kernel/pipeline.py demotes to a boot shim.
• Security: Governance deny-by-default via PrologGate ✓. Audit D5 correctly scoped SPEC-099 down to JWT+PAM++ for a single-operator private Incus deployment — matches the WAF principle "don't build controls a threat model doesn't require." No action.
• Operational Excellence: the compliance no-op (G3) and shadow verification (G2) are invisible today — both should emit metrics so "gate present but not blocking" is observable, not surprising. Audit D4 already pruned dead OTel/structlog deps; re-add with wiring when SPEC-100 observability is scoped (S3).
• Performance Efficiency: free-threaded 3.14t gives real parallelism for the SwarmCoordinator (max-5 juniors) and provider fan-out — the headline capability. No concern.
• Cost: N/A (self-hosted Incus, single operator) — the one genuine cost driver is LLM tokens; the fixed pipeline + MAX_QA_ROUNDS=4 / MAX_SELF_REVIEW_CYCLES=3 caps bound it. No action.
• Antipattern sweep: no Busy-Database / Chatty-I/O smells in the critical path (CXP is Redis-batched). The one latent antipattern is "silent gate" (G2/G3) — a control that looks active but passes everything; cheaper to make observable now than to discover in a dogfood post-mortem.

4. Decision records (repo convention — plans/decisions-*.md D-numbering; append-only)

The repo records decisions as D-numbered entries (decisions-2026-07-06.md, spec-audit/01-DECISIONS.md), not docs/adr/. These follow that convention and are one-way-door calls for Mohamed to ratify.

• D-A — Verification gate rollout: keep enforce=False (shadow) until shadow_block_rate is measured over N real intents; then flip per-stage (Cd/Qa first) behind VerificationGateConfig. Reversible per-stage; ratify the flip criterion.
• D-B — PAMEnforcer: strangle into PrologGate (port any unique capability) then quarantine the duplicate to experimental/. One-way for the file move; the capability is preserved.
• D-C — Silent gates observability: G2/G3 must emit "computed verdict / would-block" metrics before any enforce flip. Reversible; low cost.
• D-D — ToolSearchTool exposure: wire to AGENT_TOOLS for U:BA/U:Ar/U:Cd only. Reversible.

5. Explicitly NOT recommended

• No deletion of any spec-mandated capability. Every gap above is a wire, a config flip, or a duplicate-quarantine. Unimplemented FRs (007b, 022, the remaining PARTIALs) are backlog to build, tracked by the existing S1–S10 swarm plan — not candidates for removal.
• No re-architecture: the layered monolith, fixed pipeline, CXP-only comms, and two-graph topology are sound and self-consistent; the work is finishing the wiring, which is exactly what the codebase's own program is doing.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks