core package

Reviewed against commit 8a2cb42ad on 2026-10-23.

Reference: package page (92 modules, 18,404 lines, fifteen subpackages) · key modules: graph_factory, redis_factory, infra_config, env_bootstrap, prolog_adapter, artifact_store, packet, graph, protocols, threading. The layers core sits under are explained in the overview.

The story

Every large system needs a place where its shared vocabulary lives, and in orchestrata-ultimate that place is orchestrata_ultimate.core. Its package docstring says it plainly: "Orchestrata Core — interfaces, enums, traceability, config, domain, state, adapters." This is the foundation layer that the other packages (about thirty top-level packages in all, at commit 8a2cb42ad) stand on. It exists because orchestrata-ultimate was ported from a 26-package monorepo into a single package: instead of each subsystem carrying its own base classes, config readers, and infrastructure clients, all of that was consolidated here so the kernel, postman, orchestrator, gateway, memory engine, and the volk agents could all be built against one common substrate without importing each other.

The package solves two coupling problems at once. Structurally, it supplies the base classes every generated or ported component inherits — BaseEntity/DomainEntity for identity-based domain objects, BaseError for structured exceptions, BaseService for the IComponent lifecycle, and BaseRepository[T] for async CRUD. Behaviorally, it defines the Protocol contracts that let subsystems talk without depending on each other's implementations: protocols/components.py declares what a kernel, postman, orchestrator, and state manager look like; protocols/agent.py holds VolkProtocol, the contract every volk agent satisfies; and protocols/decoupling.py exists purely to break import cycles (Prolog bridge, volk registry, audit trail, Redis state). When the kernel dispatches work to a volk, or the pipeline runner asks the memory engine to recall context, both sides of the call are typed against a core protocol rather than a concrete class.

A typical piece of work flowing through the pipeline touches core at nearly every step even though core never orchestrates anything itself. Configuration is resolved through infra_config.py (Pydantic Settings) and config/manager.py; connections to the real infrastructure — Redis, the operational graph (orchkg on orchgraph, a Memgraph fork; Neo4j was retired on 2026-07-27), SWI-Prolog — come from redis_factory.py, graph_factory.py, and prolog_adapter.py, all written to be safe under free-threaded Python 3.14t. Log lines carry correlation IDs from logging.py; execution can be recorded as Spans by the sys.monitoring-based engine in monitoring/; and per-iteration pipeline debugging goes through the run-scoped tracer registry in tracing/context.py. The adapters/claude/ subpackage adds the Claude Code integration surface — hook orchestration and the ToolInterceptor that lets governance veto tool calls.

Two more concerns are deliberately made first-class here. Requirement traceability — the @traces decorator, the RTM data model, and the AST scanner that extracts annotations without importing modules — lives in traceability/, traceability_utils.py, and tracing/scanner.py; the dependency diagram below shows nearly every module pointing at traces, because virtually all of core is itself annotated. And threading/ hosts the project's parallel-execution machinery, including the batched test executor that replaces the banned pytest-xdist. Neighboring packages consume core in one direction only: agents, kernel, orchestrator, postman, memory, and providers all import from core, while core imports from none of them.

Key responsibilities

• Foundation base classes — BaseEntity/DomainEntity, BaseError, BaseService (the IComponent lifecycle), and the BaseRepository[T] async-CRUD protocol that everything generated or ported inherits.
• Subsystem contracts — the Protocol/interface layer (VolkProtocol, ComponentProtocol and its kernel/postman/orchestrator/state descendants, plus the cycle-breaking decoupling protocols) that lets packages call each other by contract, not by concrete class.
• Infrastructure access — one funnel to the outside world: infra_config.py (Pydantic Settings), config/manager.py, and thread-safe factories for Redis, the operational graph (Bolt), and the janus-swi PrologEngine, all safe under free-threaded 3.14t.
• State & domain — the persistent StateManager with namespaces, and the shared domain enums and portfolio-goal models.
• Observability — structured logging with correlation IDs, the sys.monitoring-based TracerEngine/Span model, and the run-scoped per-iteration pipeline tracer.
• Requirement traceability — the @traces decorator, the RTM data model, and the import-free AST scanner used to build the traceability matrix.
• Parallel execution — the ThreadPool/ThreadPoolManager primitives and the batched, JUnit-parsing test executor that stands in for the banned pytest-xdist.
• Claude Code integration — hook orchestration and the governance-aware ToolInterceptor.

Class diagram

Foundation base classes and state

The classes almost everything else inherits from. BaseEntity gives identity and serialization; BaseError structures every exception; BaseService and StateManager both realize the IComponent lifecycle; BaseRepository[T] is the async-CRUD contract for persistence (generated from the real source, render-verified):

Subsystem contracts (protocols/)

The Protocol layer that keeps the single-package port from collapsing into a tangle. VolkProtocol is the contract every volk agent satisfies; the top-level component contracts all descend from a common ComponentProtocol so subsystems reference each other by contract alone (render-verified):

Infrastructure, threading, and observability

Configuration and connection factories, the in-house parallel-execution machinery, and the runtime-monitoring model. ThreadPoolManager tracks ThreadPoolTasks; the Claude ToolInterceptor records Spans into the monitoring engine (render-verified):

Module dependency diagram

Internal module imports within orchestrata_ultimate.core (generated from the AST, render-verified):

base/ — foundation classes

The base/ subpackage is where the "everything inherits from here" story begins. BaseEntity (base/entity.py) is the identity-based domain object carrying a UUID and created/updated timestamps, together with recursive (de)serialization helpers, and DomainEntity extends it for the domain layer. BaseError gives every subsystem structured exceptions with a code, message, and context payload instead of bare strings, while BaseService implements the IComponent lifecycle with config and logging so services across the codebase start, stop, and report health the same way. BaseRepository[T: BaseEntity] is a typing Protocol defining async CRUD, which means persistence backends can be swapped without touching the callers. Everything generated or ported into the codebase builds on this layer, which is why the class diagram above shows so much fanning out from these few names.

protocols/ — subsystem contracts

The Protocol layer is what keeps the single-package port from collapsing into a tangle. protocols/base.py defines the shared vocabulary — ComponentType, HealthStatus, ComponentMetadata, SystemEvent, EventBusProtocol — and protocols/components.py builds the top-level component contracts on it: OrchestrationKernelProtocol, PostmanProtocol, OrchestratorProtocol, and StateManagerProtocol, all descending from a common ComponentProtocol. protocols/decoupling.py exists specifically to break import cycles, hosting GovernancePrologBridgeProtocol, VolkRegistryProtocol, AuditTrailProtocol, and RedisStateManagerProtocol so that, for example, governance and the agent runtime can reference each other by contract alone. protocols/agent.py holds VolkProtocol, the contract every volk agent satisfies (interfaces/volk.py's IVolk is a thin alias of it), while protocols/memory.py (SPEC-010) and protocols/providers.py (SPEC-011) carry the memory-engine and LLM-provider data types and protocols. The parallel interfaces/ subpackage mirrors these as the compatibility interface layer kept from the original monorepo.

Infrastructure — infra_config.py, redis_factory.py, graph_factory.py, prolog_adapter.py

All connections to the outside world are funneled through this quartet. infra_config.py centralizes infrastructure configuration through Pydantic Settings — InfraConfig for the shared services, per-volk VolkConfig with provider/model resolution helpers, and a validate() that checks the environment up front rather than failing mid-run. With configuration resolved, redis_factory.py hands out thread-safe sync and async Redis clients designed for free-threaded Python 3.14t, including pool cleanup, and graph_factory.py (until 2026-07 neo4j_factory.py) does the same for the operational-graph Bolt drivers (create_graph, create_async_graph) with URI, database, dialect and auth resolution from KG_ROUTING_URI / KG_URI, KG_DATABASE and KG_ENGINE; the neo4j import there is only the Bolt driver package. Under RUL-003, infra_env_rule.py makes unset infrastructure keys raise instead of falling back to a literal, and env_bootstrap.py is the canonical two-phase environment loader (RUL-019) that entry points call first. prolog_adapter.py wraps the janus-swi SWI-Prolog engine in PrologEngine — which keeps a per-thread engine via threading.local() — and is what the governance gates consult when they evaluate policy rules. Because every other package obtains its clients here instead of constructing them ad hoc, connection behavior is consistent — and testable — across the whole system.

graph/ — operational-graph repository seam

The graph/ subpackage (backlog #163) routes direct operational-graph access (orchkg; formerly Neo4j) through a repository/adapter seam so callers depend on a Protocol, not on the driver. graph/interfaces.py defines the UnitOfWork, SyncGraphRepository, and AsyncGraphRepository Protocols; graph/config.py holds GraphRepositoryConfig; and graph/repository.py provides the single entry points create_graph_repository / create_async_graph_repository. The concrete BoltSyncRepository and BoltAsyncRepository adapters (bolt_sync.py, bolt_async.py; until 2026-07 named for Neo4j) live under graph/adapters/, so the backing store can be swapped without inheritance. graph/retry.py adds a bounded, jittered retry for the MVCC write conflicts that Memgraph raises where Neo4j used to block on locks; both adapters use it.

threading/ — free-threaded execution and the no-xdist test runner

The threading/ subpackage carries the project's answer to a hard local constraint: pytest-xdist deadlocks this suite, so the runner had to be built in-house. thread_pool.py supplies ThreadPool, ThreadPoolManager, ThreadPoolTask, and a module-level get_default_thread_pool. test_executor.py builds on that pool to batch test modules into pytest subprocess runs, parse the resulting JUnit XML, and attribute pass/fail results back to individual modules — this is the core of the mandated scripts/run_tests.py runner. Alongside it, pytest_live_observer.py is a pytest plugin that streams per-test events as SIGKILL-durable NDJSON and maintains a session registry, so external observers such as scripts/observe_pytest.py can attach to a live run in real time; the dependency diagram shows test_executor consuming both the observer and the pool. Since the original write-up the subpackage has gained pytest_live_reader.py (readers for the observer's registry and event log), failure_detail.py (full tracebacks, locals and captured output for failing tests, scrubbed before they are kept) and test_scheduler.py (a CLP(FD) scheduler that places test modules on workers through a Prolog module).

Traceability — traceability/, traceability_utils.py, tracing/

Requirement traceability is treated as a first-class concern rather than an afterthought, and the dependency diagram makes that visible: almost every module in the package imports traces. traceability_utils.py defines the @traces decorator and the data model — TraceInfo, Requirement, UserStory, RTMReport — while traceability/__init__.py collects traces from source trees and generates the requirements traceability matrix in both JSON and markdown. tracing/scanner.py complements them with an AST scanner that extracts @traces annotations without importing the modules, so the RTM can be built even for code that will not import cleanly. A separate concern lives in the same directory: tracing/tracer.py plus tracing/context.py implement the run-scoped, config-gated per-iteration runtime tracer registry keyed by pipeline thread_id, used to debug pipeline behavior; tracing/traceability.py is a superseded thin shim retained for compatibility.

Observability — logging.py and monitoring/

When something goes wrong in a multi-agent pipeline, core's observability layer is what makes the failure legible. logging.py provides centralized structured logging with correlation IDs — StructuredFormatter emits JSON for machines, StandardFormatter renders for humans, and configure_logging/get_logger/new_correlation_id tie a request's log lines together across subsystems. For deeper introspection, monitoring/engine.py implements a TracerEngine on the sys.monitoring API introduced in Python 3.12, exposed through a monitor entry point, recording execution as Span objects defined in monitoring/spans.py with func_name, entry/exit nanosecond timestamps, computed duration_ms, and a SpanStatus (OK/ERROR/TIMEOUT). The Claude Code ToolInterceptor in adapters/claude/ feeds this same engine, which is why the dependency diagram shows tool_interceptor --> engine.

Module inventory

Class map (generated)

Generated from the live source AST by scripts/gen-module-class-diagrams.py — do not hand-edit. Abstract classes have a green border; members listed are those defined on the class itself (inherited members excluded). Solid empty-triangle arrows = in-package inheritance; dashed arrows into dashed boxes = external/stdlib/cross-package bases; filled diamond = composition (constructed in __init__); open diamond = aggregation (annotated/injected reference); diamond edge labels name the owning attribute.