kernel package

Reviewed against commit 8a2cb42ad on 2026-10-23.

Reference: package page (51 modules, 19,600 lines; subpackages audit, pam and performance) · key modules: kernel, router, dispatcher, pipeline, policy_evaluator, cli_main. The kernel's place in a request's journey is described in the overview. Note: VolkRegistry no longer lives in this package; it is now agents.volk_registry.

The story

orchestrata_ultimate.kernel is the Orchestration Kernel — the central coordination engine of orchestrata-ultimate. Every other subsystem in the project is a specialist: cxp defines the packet protocol, agents implements the volks, governance holds the Prolog policy machinery, knowledge_graph persists what happened. The kernel is the piece that makes them behave like one system. It owns component lifecycle and service discovery through OrchestrationKernel (with OrchestrataKernel as its subclass alias), publishes system events over a dual-layer EventBus that works in-process and, when Redis is available, across processes, and provides the shared services everything else depends on: message routing, durable queuing, session management, policy enforcement, audit logging, and performance primitives.

A message's journey through the kernel starts as a CXP packet. CXPRouter decides where it should go — unicast to a specific volk, or fan-out — consulting the VolkRegistry (the registration-and-discovery directory of every agent, with capabilities, health, and metadata) to resolve the destination. MessageDispatcher then performs the actual protocol-specific delivery (CXP over the cxpt direct transport, WebSocket, HTTP; the dispatcher's grpc function and key names are historical, grpcio having been retired on 2026-08-07), keeping per-attempt retry accounting, and can consult a staged, audit-only PolicyEvaluator before delivering. If the target volk is offline, QueueManager parks the packet in a Redis-backed durable queue with stale-message expiry, so nothing is silently lost. Governance is enforced in depth rather than at a single choke point: policy_evaluator.py evaluates routing, permission, and ownership decisions against real Prolog rules, while the pam/ subpackage implements PAM++ (Persona-Assignment-Mapping) enforcement — PAMEnforcer checks authority, ownership, boundary, and permission status, and the escalation managers walk multi-hop escalation chains when a check fails. Every consequential decision can be recorded through the audit/ subsystem, which buffers typed AuditEntry records and flushes them to file or graph writers (GraphAuditWriter, writing to the operational graph orchkg; it was Neo4jAuditWriter until the 2026-07 migration), with a Redis queue for durable fire-and-forget logging.

The kernel is also where the product-level flow lives: PipelineRunner in pipeline.py boots the built-in volk pipeline (Concierge through Compliance; its docstring now says 11 volks, earlier text here said 9) and routes user commands through it. It is deliberately composed from mixins so the runner stays comprehensible — PipelineSessionMixin handles session lifecycle, PipelineKGMixin writes stage state and injects context from the knowledge graph, and PipelineEscalationMixin routes escalations back to a human. Around the runner sit its support cast: pipeline_events.py broadcasts stage-transition milestones to Redis so UIs can watch a run live, pipeline_artifacts.py keeps a queryable on-disk record of stage outputs, and pipeline_checkpoint.py lets an interrupted run resume by reusing completed LLM-stage outputs rather than re-prompting (LLM stages are not replayable, so checkpointing is the only safe resume). pipeline_cli.py is the interactive REPL front-end over all of this, and cli.py/cli_main.py are the process entry points — the latter boots the full monolithic deployment: kernel, postman, intent pipeline, agent runtime, and health server.

In the neighborhood of packages, the kernel sits at the hub. It consumes cxp packets and endpoint configuration, dispatches into agents volks (including plugin-registered ones via _PluginVolkAdapter), leans on governance for the Prolog gate, persists to knowledge_graph and the session controller, and delegates actual message delivery to postman. Cross-cutting concerns — Redis and graph-driver factories, @traces requirement decorators, infra config — come from core. The remaining modules round out kernel services: session_manager.py/session_context.py for session persistence and routing, config_manager.py for versioned typed configuration, resource_monitor.py for CPU/memory/disk alerts, and the performance/ subpackage for caching, pooling, batching, clustering, autoscaling, SLA, and chaos-testing primitives.

Key responsibilities

• Component lifecycle & service discovery — OrchestrationKernel initializes, starts, stops, registers/unregisters components, and discovers service endpoints, so callers hold one kernel instead of a bag of services.
• System event fan-out — the dual-layer EventBus delivers SystemEvents to in-process subscribers and, when Redis is present, across processes, with replay from a Redis stream.
• CXP message plane — CXPRouter resolves recipients (unicast / fan-out) via the VolkRegistry, MessageDispatcher delivers over CXP (cxpt direct transport)/WebSocket/HTTP with retry accounting, and QueueManager durably parks packets for offline volks with stale-message expiry and dead-lettering.
• Volk registration & discovery — VolkRegistry is the directory of every agent, queryable by alias, capability, and protocol, with heartbeat/health tracking and registration notifications.
• Governance enforcement — PolicyEvaluator (routing/permission/ownership/escalation against real Prolog) and the pam/ PAM++ subsystem (PAMEnforcer + escalation managers) enforce authority, ownership, and boundaries.
• The built-in volk pipeline — PipelineRunner (Concierge → Compliance; 11 volks per its docstring) boots and drives runs, composed from session, knowledge-graph, audit and escalation mixins, with milestone events, artifact capture, and checkpoint/resume.
• Audit trail — AuditLogger buffers typed AuditEntry records and flushes to file / graph writers, with a Redis queue for durable fire-and-forget logging.
• Kernel services — session persistence (SessionManager/SessionContext), versioned typed configuration (ConfigManager), resource monitoring (ResourceMonitor), and the performance/ scalability toolbox.
• Process entry points — cli.py (standalone kernel) and cli_main.py (monolithic deployment: kernel + postman + intent pipeline + agent runtime + health server).

Class diagram

The principal classes, with their real attributes and methods, grouped by cluster (each block render-verified).

CXP message plane

OrchestrationKernel owns the EventBus; CXPRouter resolves recipients through the VolkRegistry and hands delivery to MessageDispatcher, which parks undeliverable packets in QueueManager and consults a staged PolicyEvaluator in audit-only mode.

Governance enforcement

Both governance engines answer against real Prolog. PolicyEvaluator decides routing/permission/ownership and builds escalation paths; PAMEnforcer enforces authority, ownership, boundary, and permission and records violations.

The built-in volk pipeline

PipelineRunner is composed from four mixins (PipelineSessionMixin, PipelineKGMixin, PipelineAuditMixin, PipelineEscalationMixin) and owns its support cast — the event bus, artifact store, and checkpoint store — so an interrupted run can resume without re-prompting completed LLM stages. The diagram below was drawn before PipelineAuditMixin was added and does not show it.

Kernel services

Three long-lived service objects round out the kernel: session persistence across Redis + the operational graph, versioned typed configuration with rollback, and system resource monitoring with threshold alerts.

Inheritance hierarchy

Class inheritance within orchestrata_ultimate.kernel (generated from the AST, render-verified) — note the mixin composition of PipelineRunner (four at commit 8a2cb42ad; the diagram predates PipelineAuditMixin), the OrchestrationKernel → OrchestrataKernel alias, and the pluggable AuditLogWriter writers.

Module dependency diagram

Internal module imports within orchestrata_ultimate.kernel (generated from the AST, render-verified):

Module inventory

kernel.py — OrchestrationKernel

OrchestrationKernel (implementing IOrchestrationKernel) is the coordination heart of the package. It owns component lifecycle and service discovery, and ties together the router, dispatcher, queue manager, session manager, volk registry, and audit subsystem so that the rest of the codebase interacts with one kernel object rather than a bag of services. It publishes lifecycle and system events through the EventBus, which layers in-process subscribers on top of optional Redis pub/sub so events survive process boundaries when the infrastructure is present. OrchestrataKernel is a thin subclass alias used elsewhere in the codebase.

router.py + dispatcher.py + queue_manager.py — the CXP message plane

These three modules form the kernel's message plane. CXPRouter decides where a CXP packet goes, supporting unicast and fan-out routing modes and consulting the VolkRegistry to resolve destinations; it tracks per-message MessageStatus and aggregate RoutingMetrics, returning a typed RoutingResult for each decision. Once routed, MessageDispatcher performs the protocol-specific delivery (CXP over the cxpt direct transport, WebSocket, HTTP; the code still names the CXP path dispatch_grpc) with retry accounting captured in DispatchAttempt/DispatchResult records, and wires in a staged PolicyEvaluator via _build_default_policy_evaluator() so policy checks run in audit-only mode alongside dispatch. When a target volk is unreachable, QueueManager steps in with Redis-backed durable queues — packets wait for the volk to come back online, QueueStats exposes queue health, and stale-message expiry prevents unbounded growth.

policy_evaluator.py + pam/ — governance enforcement

Governance in the kernel is real Prolog, not stubs. policy_evaluator.py evaluates routing, permission, and ownership decisions against Prolog rules, returning typed results (RoutingResult, PermissionResult, OwnershipResult, EscalationPath, DryRunResult) and emitting a PolicyAuditEntry for every decision; failures surface as PolicyEvaluationError and its PolicyTimeoutError/PolicyQueryError subclasses so callers can distinguish an engine outage from a denied query. Alongside it, the pam/ package implements PAM++ (Persona-Assignment-Mapping) enforcement: PAMEnforcer answers whether an agent has the authority, ownership, boundary clearance, and permission level to act (the enums and result dataclasses live in pam/types.py), and when a check fails, EscalationManager and MultiHopEscalationManager walk the escalation chain toward a party that can decide. pam_enforcer.py remains as a backward-compat shim over the decomposed package.

pipeline.py and its mixins — the built-in volk pipeline

PipelineRunner is the static-boot pipeline engine: it instantiates the built-in volks (the module docstring says 11; this page used to say nine), connects them through postman and the Prolog gate, and routes user commands through the pipeline stages. Rather than one monolithic class, it composes three mixins — PipelineSessionMixin manages the session lifecycle and work log, PipelineKGMixin synchronizes stage state to the knowledge graph and injects graph context into stage prompts, and PipelineEscalationMixin routes escalations out to the human operator. The supporting modules complete the picture: pipeline_events.py publishes a PipelineEvent at every stage transition to Redis so external observers (the gateway, UIs) can follow a run in real time; pipeline_artifacts.py writes each stage's outputs to a queryable on-disk ArtifactStore; and pipeline_checkpoint.py records StageCheckpoints so a resumed run reuses completed LLM-stage outputs instead of re-prompting — essential because LLM stages are not deterministically replayable. pipeline_cli.py wraps all of this in an interactive REPL with intent, respond, status, review, approve, packets, provenance, and log commands, and _PluginVolkAdapter lets plugin-registered volks participate in dispatch as first-class stages.

audit/ — audit trail

The audit subsystem gives every kernel decision a durable record. AuditLogger buffers AuditEntry records — typed by AuditEventType and AuditOutcome, configured via AuditLogConfig — and flushes them to pluggable AuditLogWriter destinations, of which FileAuditWriter and GraphAuditWriter ship in the box. For callers that cannot afford to wait on a write, RedisAuditQueue provides durable fire-and-forget queuing, so audit records survive even if the writer is temporarily behind. audit_logger.py is the backward-compat shim over the decomposed audit/ package.

performance/ — performance & scalability primitives

PerformanceManager is a facade over a toolbox of operational primitives: metric trackers (LatencyTracker, ThroughputTracker, MemoryTracker, UptimeTracker) measure how the system behaves, resource primitives (LRUCache, ConnectionPool, BatchProcessor, RingBuffer, BackpressureController) keep it behaving well under load, and the cluster layer (ConsistentHash, LeaderElection, NodeHealth, ClusterManager) plus AutoScaler handle multi-node concerns. SLA targets and thresholds live in performance/sla.py, and ChaosController supports deliberate fault injection for chaos testing. performance_manager.py is the corresponding compat shim.

Class map (generated)

Generated from the live source AST by scripts/gen-module-class-diagrams.py — do not hand-edit. Abstract classes have a green border; members listed are those defined on the class itself (inherited members excluded). Solid empty-triangle arrows = in-package inheritance; dashed arrows into dashed boxes = external/stdlib/cross-package bases; filled diamond = composition (constructed in __init__); open diamond = aggregation (annotated/injected reference); diamond edge labels name the owning attribute.