Knowledge Wiki Orchestrata Ultimate

Home / Projects / Orchestrata Ultimate

Orchestrata Ultimate

What this is. Orchestrata is a governance and agentic workflow engine for code, architecture and communications. A person states an intent; a WebSocket gateway authenticates it and hands it to the kernel over CXP. The kernel checks it against Prolog policy and routes it through specialist agents, the volks, from concierge to documentor. Every feature starts as a spec, code claims its requirements with @traces, and the RTM grades the evidence. Everything runs on real infrastructure in three isolated Incus fleets; Mohamed is the sole author and the human in the loop.

Start here

  1. Orchestrata, explained: the whole system in eleven sections, with diagrams.
  2. Starting context: the first-hour orientation, the rules that bind, and the recurring defect.
  3. The volks: who each agent role is and how work passes between them.
  4. Architecture: the six layers, the packages and the runtime pieces.
  5. Runtime services: what actually runs on the host, and why.
  6. Workflow: how a change goes from brief to production.
  7. Glossary: every term you will meet, each linked to its page.

The table of contents lists every page family in reading order.

Architecture

The system is a modular monolith with six import layers, from vocabulary up to the interface, and five runtime pieces: gateway, kernel, orchestrator, postman and the volk runtime. Under them sit Redis and the orchgraph cluster with its two databases, orchkg (operational knowledge graph) and memgraph (semantic / RAG). The monolith is being re-cut into ten component packages, which has not cut over yet.

Code reference

Every package has a generated reference page (one page per module, with class and dependency diagrams and an API table) and most have a curated narrative under modules/ that explains why the package exists. Start from the code reference index or the module docs index; the table below links both for each package. Module counts and the reference revision (8a2cb42ad) come from the reference index; layers come from architecture-standards.md.

Package Layer Modules What it is Reference Narrative
agents 3 78 The multi-agent runtime: base agent, the volks, pool, lifecycle, tool loop reference agents
context – 11 Top-level context assembly reference context
core 0–1 92 Interfaces, enums, traceability, config, domain, state, adapters reference core
cxp 2 37 The Cross-agent eXchange Protocol and its transport reference cxp
di 1 2 The dependency-injection container reference di
domaingen – 7 Ontology-driven domain model generation (SPEC-105) reference domaingen
drivers 3 12 Async code-generation drivers for the junior volks reference drivers
factory – 74 SPEC-033 governed autonomous build factory reference factory
gates – 5 The Prolog gate layer: facts from the AST snapshot, policy as rules reference gates index
gateway 5 49 The WebSocket JSON-RPC gateway (SPEC-014) reference gateway
governance 3 28 Policy engine, escalation, audit, constitutional enforcement reference governance
harness – 20 Volk test harness: scenario-driven evaluation reference harness
infra 1 3 Relocated state stores and adapters reference none yet
integrations – 4 Adapters for external APIs and services reference integrations
kernel 4 51 The central coordination engine and pipeline reference kernel
knowledge_graph 3 55 The operational KG (orchkg) client, sync and queries reference knowledge_graph
llm 3 5 The async LLM client facade for volk brains reference llm
memory 2 8 The memory engine with dual-layer storage reference memory
openspec – 7 OpenSpec format support (SPEC-104) reference openspec
orchestrator 4 42 U:Ox: intent parsing, routing, policy management reference orchestrator
planning – 2 Support for long-horizon agent execution reference planning
plugin 5 14 The plugin SDK (SPEC-015) reference plugin
postman 2 31 U:PS: message routing and delivery reference postman
promotion – 23 Deployment: what promote has to mean reference promotion
providers 3 16 The unified LLM provider abstraction reference providers
refactor – 50 Tool-calling refactor capability: codemod and rename engines reference refactor
semantic_graph – 20 The RAG / GraphRAG repository over the memgraph database reference semantic_graph
session 4 63 Session management and the session CLI reference session
tools – 20 The unified tool framework and registry reference tools
verification 3 66 The verification agent (SPEC-035) reference verification
orchestrata_ast (root) – 7 The AST analyzer behind the graph sync reference none yet
orchestrata_tools (root) – 60 The src/orchestrata_tools root reference none yet
scripts/ – 182 Runners, gates and entry-point scripts reference none yet

A dash in the Layer column means the package is not in the layer table of architecture-standards.md. The component packages under packages/ have their own reference pages: contracts, governance, knowledge_graph, kernel, session, volks, gateway and tools; their dependency edges are on the component DAG.

Workflow & gates

Work is done from written briefs: the host Claude session measures and writes the brief, a worker model (Sonnet, Qwen or DeepSeek) writes the code, and the result is checked by execution before it is committed (ADR-0125). A gate is a mechanical check with a command and no discretion; the battery, the suite preflight, the session hooks and the Prolog gate layer all say no without asking.

Deployment & operations

Prod runs from the pinned checkout /opt/orchestrata (commit 4ac0db570 on 2026-10-23) as systemd user units on the host, over the orchgraph cluster and redis-queue in the Incus project default. deploy/ship.sh decides that work is finished and deploy/deploy.sh puts it on machines; the pages below describe both, every unit, and the measured state.

Tests & reports

The suite runs against real Redis, a real graph and a real kernel in the test fleet, with no doubles, no skips and no pytest-xdist; a run that does not start and verify the kernel is not a valid run. Reports record each authoritative run, the green ledger records which commits passed, and the duration files show where the time goes.

Specs, RTM & briefs

specs/ has 79 tracked entries, 71 of them numbered spec directories; each spec defines requirement ids that code and tests cite with @traces. The RTM joins the two and grades every requirement by its strongest evidence, from TESTED down to UNLINKED. Briefs are the written tasks workers build from, with a fixed template.

Decisions & plans

One-way-door decisions are ADRs in the repo's docs/adr/ (105 files at 478d9a0ef) and are mirrored on the decisions page; standing user rulings are the wiki's rulings canon, RUL-001 to RUL-041. Plans record the design work that leads to an ADR.

Sessions, tasks & research

Every substantial work session is logged, open work items are tracked as tasks, and investigations are written up as research pages with their measurements. Start at the session index for history and the tasks page for what is open now.

Tooling (codetools, Claude Code)

Orchestrata is developed with separately deployed tools under /opt: orchestrata-tools (the AST toolkit, daemons, pytest-live, ship), codetools (code tooling as a CLI, MCP server and plugin), Hermes and pi-task (non-Claude workers). The Claude Code side is the orchestrata plugin (hooks, skills, agents, commands) and the orchestrata MCP server, which talks to the gateway.

Status history

Earlier versions of this hub carried dated status notes. They are kept here as a record; the current state is in Orchestrata, explained and on the pages above.

The kernel is required (kept from the 2026-08 hub)

scripts/start_test_services.py marks the kernel required=True alongside Redis, the graph service and the gateway. When it is down, the runner starts it (python -m orchestrata_ultimate.agents.runtime_main) and verifies it over a real channel; if start or verify fails, scripts/run_tests.py aborts. A run that does not start and verify the kernel is false-green. The only sanctioned opt-out is an explicit --skip-services kernel, and such a run is non-authoritative and must be labelled so. Source: docs/kernel-is-required.md.

State recorded on 2026-08-05

Goals (from the 2026-08 hub)

  1. Intent → Code: Reduce the distance between a human goal and merged, tested implementation.
  2. Spec-Driven Development: Every feature starts from an approved spec and is traced through code (@traces) to tests.
  3. Multi-Agent Collaboration: Specialist agents (volks) argue, review, and refine work like a senior engineering team.
  4. Auditability: Every decision, change, and failure is logged and traceable.
  5. Real Infrastructure: Real LLMs, databases, browsers, shells — no mocks, no monkeypatch (suite-wide).
  6. Cross-Model Resilience: Multiple provider backends with runtime profile switching.

2026-08-05 — dispatch-reply & pipeline-event work

Guided reading & stories (from the 2026-08 hub)

Quick commands (from the 2026-08 hub)

Current commands select the test profile first (ORCHESTRATA_ENV_PROFILE=test, AGENTS.md); start_test_services.py now also starts the kernel.

python scripts/start_test_services.py   # start + verify Redis / Graph / Gateway / Kernel
python scripts/run_tests.py             # full test suite (~10 min, batched)
python scripts/ingest_orchestrata_ultimate.py     # ingest source into the semantic graph
python scripts/enrich_semantic_graph_specs.py     # enrich specs + @traces

Key rules (from CLAUDE.md, 2026-08)

Orchestrata Ultimate is an active research and engineering project. The goal is not to replace engineers, but to give every engineer a tireless, verifiable, multi-agent team.

Packet 2026-10-18 — SPEC-023 goal system (evaluate, Stop gate, alarm, injection)

What changed: GoalMixin.goal_evaluate now runs every exit criterion's verify_command/verify_pattern and grades it (pending/met/failed transitions logged as progress entries; all-met → achieved + success alarm). The US-005 Stop gate is wired into SessionController.end() — an active, incomplete, unblocked goal refuses the end (GoalNotAchievedError with the "Goal not achieved. Remaining: …" injection); a blocked goal ends with the blocker reason + attention alarm; skip_goal_check is the escape hatch. New session/audio_alarm.py (AudioAlarm): ascending success tones / low attention tones, backend chain ffplay→paplay→aplay→mpv→bell with real synthesized WAVs, non-blocking, 30s repeat until stop(). Session start now injects the active goal + criteria status (session_auto_start_hook). CLI: session goal block/unblock/evaluate; achieve/block trigger their alarms.

Where: src/orchestrata_ultimate/session/{audio_alarm.py, controller/_goals.py, controller/_core.py, exceptions.py, cli/goals.py, cli/main.py, hooks/session_auto_start_hook.py, event_emitting_session_controller.py}; checklist created at specs/023-goal-system/checklists/requirements.md (REQ-023-007..038; US-006 re-planning 027..031 recorded open).

Tests: tests/session/test_goal_evaluate.py (12), test_goal_stop_gate.py (6), test_audio_alarm.py (11), hooks injection (+4). Full session suite 105/105 modules / 2502 tests green; _goals.py 97%, audio_alarm.py 94% coverage. Commit: a358de7e14b5a50ab7d51a6d38c75416119f3c3e.