Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate / Specs / 010-memory-engine

010-memory-engine: Memory Engine

The Memory Engine is the unified agent memory abstraction for Orchestrata. It provides store, retrieve, search, and summarize operations over a dual-layer storage system (Redis hot / Neo4j cold), enabling Volks to maintain working memory within sessions and long-term memory across sessions without exceeding LLM context windows.

At a glance

Item Value
Spec directory specs/010-memory-engine/
Status line in the spec Draft
Requirement IDs defined 55
With a @traces citation in src/ 53 (96.4%)
With a @traces citation in tests/ 5 (9.1%)
Tracked files 9
RTM page 010-memory-engine

Reading this spec

The Memory Engine is the unified agent memory abstraction for Orchestrata. It offers store, retrieve, search and summarize operations over a dual-layer storage system, so that Volks keep working memory within a session and long-term memory across sessions without exceeding LLM context windows. The spec exists because the knowledge graph (SPEC-008) stores code structure rather than agent reasoning, session management (SPEC-006) tracks session state rather than working memory, and no unified interface existed for agents to store and retrieve contextual memory.

The scope is six user stories: working memory storage, cross-session memory and context window management (priority 1), and semantic search, memory summarization and shared team memory (priority 2). Requirement IDs use the REQ-010- prefix.

The key decisions are an IMemoryEngine interface registered in the DI container as ComponentType.MEMORY_ENGINE; Redis for hot working memory with operations under 50ms and Neo4j for cold long-term memory; embedding-based semantic search through sentence-transformers; token-aware context compression to fit provider context windows; CXP for memory-related inter-agent communication; and PAM++ governance on every memory operation.

In the code, the largest tracers are orchestrata_ultimate/core/protocols/memory, orchestrata_ultimate/memory/hot_store, orchestrata_ultimate/memory/memory_engine, orchestrata_ultimate/memory/cold_store and orchestrata_ultimate/memory/compressor. The per-requirement view is on the RTM page.

The spec's status line reads Draft. It tracks 9 files and defines 55 requirement IDs. The full text follows below.

Measured coverage is 53 of 55 requirements cited by at least one @traces in src/ (96.4%), spread over 10 modules; the module citing the most of them is orchestrata_ultimate/core/protocols/memory (19 requirements). In tests/, 5 requirements are cited by a test-side @traces.

Coverage gaps: 2 requirements have no citing symbol. The largest untraced groups are REQ-010-* (2). Untraced means no @traces names the ID; the behaviour may still exist in code that was never annotated. Test-side citations are thinner than code-side ones (5 against 53).

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID Done Requirement Traced by
REQ-010-001 no IMemoryEngine defined as ABC in orchestrata-core interfaces module 1
REQ-010-002 no IMemoryEngine registered with ComponentType.MEMORY_ENGINE in DI container 5
REQ-010-003 no IComponent lifecycle implemented (initialize, start, stop, get_health) 6
REQ-010-004 no MemoryEntry data class defined with key, content, tags, metadata, classification, embedding 2
REQ-010-005 no MemoryScope data class defined with session, task, volk, project, portfolio filters 1
REQ-010-006 no Store accepts content with optional key, tags, metadata, classification 3
REQ-010-007 no Auto-generated UUID4 keys when key not provided 2
REQ-010-008 no Embeddings computed on store for semantic search 1
REQ-010-009 no Hot storage (Redis) written immediately on store 2
REQ-010-010 no Cold storage (Memgraph) written asynchronously on store 1
REQ-010-011 no Retrieve by exact key returns MemoryEntry or None 3
REQ-010-012 no Retrieve by tag filter returns matching entries 1
REQ-010-013 no Retrieve by session_id and/or task_id supported 1
REQ-010-014 no Returned entries include metadata (timestamp, volk_id, relevance_score) 1
REQ-010-015 no Semantic search via embedding similarity implemented 4
REQ-010-016 no Hybrid search (keyword + semantic) supported none
REQ-010-017 no Scope filters (session, task, volk, project, portfolio) accepted 3
REQ-010-018 no Results ranked by combined relevance score 1
REQ-010-019 no Configurable result limit enforced (default: 10) 3
REQ-010-020 no Summarize accepts context_id and max_tokens budget 2
REQ-010-021 no Memories selected by relevance to current context 1
REQ-010-022 no Compressed output fits within token budget 2
REQ-010-023 no Structured summary includes source references 2
REQ-010-024 no Working memory stored in Redis with session-duration TTL 2
REQ-010-025 no Redis hash structures keyed by session_id:volk_id 1
REQ-010-026 no Pub/sub for memory change notifications none
REQ-010-027 no Embedding vectors stored in Redis for fast similarity search 2
REQ-010-028 no Long-term memories persisted as Memory nodes in Neo4j 2
REQ-010-029 no Memory nodes linked to Session, Volk, Task, Project nodes 2
REQ-010-030 no Temporal queries supported (date range filters) 1
REQ-010-031 no Graph traversal for related memories supported 1
REQ-010-032 no Working memories migrated to cold storage on session end 4
REQ-010-033 no Cold memories promoted to hot storage on context match 1
REQ-010-034 no Expired hot storage entries garbage-collected 1
REQ-010-035 no Memory lineage preserved (summary -> original chain) 1
REQ-010-036 no Embeddings computed via sentence-transformers (all-MiniLM-L6-v2 default) 1
REQ-010-037 no Embedding cache prevents recomputation 1
REQ-010-038 no Configurable embedding model supported 1
REQ-010-039 no Cosine similarity search implemented 3
REQ-010-040 no Approximate nearest neighbor for large memory sets 1
REQ-010-041 no Token counting via provider-specific tokenizers (tiktoken, anthropic, word-based) 6
REQ-010-042 no Per-request token budgets enforced for memory injection 2
REQ-010-043 no Memory prioritization: explicit relevance > semantic similarity > recency 1
REQ-010-044 no Remaining budget returned after injection 1
REQ-010-045 no PAM++ ownership enforced on all memory operations 1
REQ-010-046 no Prolog governance queried for memory access permissions 1
REQ-010-047 no All memory operations logged to audit trail 1
REQ-010-048 no Delegation chains supported for shared memory access 1
REQ-010-049 no Memory classification (private, shared, system) enforced 2
REQ-010-050 no Reclassification supported with audit logging 2
REQ-010-051 no Hot store < 10ms (P95) 1
REQ-010-052 no Hot retrieve < 5ms (P95) 1
REQ-010-053 no Hot search < 50ms (P95) 1
REQ-010-054 no Cold retrieve < 50ms (P95) 1
REQ-010-055 no Cold search < 200ms (P95) 1

Code that traces this spec

Module Symbols Requirements
orchestrata_ultimate/core/protocols/memory 14 19
orchestrata_ultimate/gateway/handlers/memory 5 7
orchestrata_ultimate/gateway/rpc 1 1
orchestrata_ultimate/gateway/server 1 1
orchestrata_ultimate/memory/cold_store 10 12
orchestrata_ultimate/memory/compressor 6 8
orchestrata_ultimate/memory/embedding 6 7
orchestrata_ultimate/memory/hot_store 15 15
orchestrata_ultimate/memory/memory_engine 11 12
orchestrata_ultimate/memory/token_counter 3 1

Checklists

checklists/constitutional.md

Memory Engine Constitutional Checklist

Security and Governance Principles
Data Integrity and Consistency
Privacy and Data Protection
System Reliability and Resilience
Ethical and Responsible AI
Compliance and Regulatory Requirements
Performance and Resource Management
Interoperability and Standards
Transparency and Explainability
Innovation and Future-Proofing
Human-AI Collaboration

checklists/requirements.md

The requirement list above comes from this file; 0 of 55 checkbox items are ticked.

checklists/testing.md

Memory Engine Testing Checklist

Core Interface Testing
Store Operation Testing
Retrieve Operation Testing
Search Operation Testing
Summarize Operation Testing
Storage Migration Testing
Embedding Testing
Governance Testing
Performance Testing
Resilience Testing
Integration Testing
Edge Cases

Other tracked files

Full spec text

Verbatim from specs/010-memory-engine/spec.md; headings are pushed down two levels.

SPEC-010: Memory Engine

Metadata

Overview

What

The Memory Engine is the unified agent memory abstraction for Orchestrata. It provides store, retrieve, search, and summarize operations over a dual-layer storage system (Redis hot / Neo4j cold), enabling Volks to maintain working memory within sessions and long-term memory across sessions without exceeding LLM context windows.

Why
How

User Scenarios & Stories

US-001: Working Memory Storage (P1)

As a Volk I want to store intermediate reasoning and decisions during a task So that I can reference them later in the same session without re-deriving them

Acceptance Criteria:

US-002: Cross-Session Memory (P1)

As a Volk I want to access decisions and learnings from previous sessions So that I build on prior work instead of starting from scratch

Acceptance Criteria:

US-003: Context Window Management (P1)

As the Kernel I want to inject relevant memories into Volk context without exceeding token limits So that Volks operate within provider context windows

Acceptance Criteria:

US-004: Semantic Search (P2)

As a Volk I want to search memories by meaning rather than exact keywords So that I find relevant context even when terminology differs

Acceptance Criteria:

US-005: Memory Summarization (P2)

As the system I want to compress verbose memory entries into summaries So that long-running sessions don't exhaust context budgets

Acceptance Criteria:

US-006: Shared Team Memory (P2)

As a team of Volks I want to share relevant memories across the team So that coordination happens through shared context, not just CXP messages

Acceptance Criteria:

Functional Requirements

Core Interface

FR-001: IMemoryEngine Interface (P1)

FR-002: Store Operation (P1)

FR-003: Retrieve Operation (P1)

FR-004: Search Operation (P1)

FR-005: Summarize Operation (P2)

Storage Layer

FR-006: Hot Storage (Redis) (P1)

FR-007: Cold Storage (Neo4j) (P1)

FR-008: Storage Migration (P1)

Embedding & Search

FR-009: Embedding Generation (P1)

FR-010: Similarity Search (P2)

Context Management

FR-011: Token Counting (P1)

FR-012: Context Budget (P1)

Governance

FR-013: PAM++ Integration (P1)

FR-014: Memory Classification (P1)

FR-015: Temporal Decay (P2)

FR-016: Update Operation (P1)

FR-017: Content Size Limits (P1)

Non-Functional Requirements

Performance

NFR-001: Hot Storage Latency

NFR-002: Cold Storage Latency

NFR-002b: Summarization Latency

NFR-003: Throughput

Reliability

NFR-004: Data Durability

NFR-005: Availability

Scalability

NFR-006: Storage Limits

Security

NFR-007: Access Control

Technical Design

Architecture
+-------------------------------------------------------------+
|                    Memory Engine System                       |
+-------------------------------------------------------------+
|  +--------------+  +--------------+  +--------------+        |
|  |   Memory     |  |   Context    |  |   Embedding  |        |
|  |   Store      |  |   Compressor |  |   Service    |        |
|  +--------------+  +--------------+  +--------------+        |
|         |                 |                 |                 |
|         v                 v                 v                 |
|  +-----------------------------------------------------+    |
|  |              Dual Storage Layer                       |    |
|  |  +-----------------+  +-------------------------+    |    |
|  |  |   Redis (Hot)   |  |     Neo4j (Cold)        |    |    |
|  |  |  Working memory |  |   Long-term memory      |    |    |
|  |  |  Embeddings     |  |   Memory graph nodes    |    |    |
|  |  |  TTL-based      |  |   Temporal queries      |    |    |
|  |  +-----------------+  +-------------------------+    |    |
|  +-----------------------------------------------------+    |
+-------------------------------------------------------------+
Key Interfaces
class IMemoryEngine(IComponent, ABC):
    """Unified agent memory abstraction."""

    @abstractmethod
    async def store(
        self,
        content: str,
        *,
        key: str | None = None,
        tags: list[str] | None = None,
        metadata: dict | None = None,
        classification: MemoryClassification = MemoryClassification.PRIVATE,
    ) -> MemoryEntry:
        """Store a memory entry. Returns the created entry with generated key."""

    @abstractmethod
    async def retrieve(
        self,
        key: str,
        *,
        include_metadata: bool = True,
    ) -> MemoryEntry | None:
        """Retrieve a memory entry by exact key."""

    @abstractmethod
    async def search(
        self,
        query: str,
        *,
        scope: MemoryScope | None = None,
        limit: int = 10,
        min_similarity: float = 0.7,
    ) -> list[MemoryEntry]:
        """Semantic search over memory entries."""

    @abstractmethod
    async def summarize(
        self,
        context_id: str,
        *,
        max_tokens: int = 2048,
        strategy: SummarizationStrategy = SummarizationStrategy.RELEVANCE,
    ) -> MemorySummary:
        """Compress memories for context_id into a token-bounded summary."""

    @abstractmethod
    async def update(
        self,
        key: str,
        *,
        content: str | None = None,
        tags: list[str] | None = None,
        metadata: dict | None = None,
        classification: MemoryClassification | None = None,
    ) -> MemoryEntry:
        """Update an existing memory entry. Only provided fields are modified."""

    @abstractmethod
    async def delete(self, key: str) -> bool:
        """Delete a memory entry by key."""

    @abstractmethod
    async def migrate_to_cold(self, session_id: str) -> int:
        """Migrate all hot memories for session to cold storage. Returns count."""


class MemoryClassification(Enum):
    PRIVATE = "private"
    SHARED = "shared"
    SYSTEM = "system"


class MemoryScope:
    session_id: str | None = None
    task_id: str | None = None
    volk_id: str | None = None
    project_id: str | None = None
    portfolio_id: str | None = None
    date_from: datetime | None = None
    date_to: datetime | None = None


class MemoryEntry:
    key: str
    content: str
    summary: str | None
    tags: list[str]
    metadata: dict
    classification: MemoryClassification
    embedding: list[float] | None
    created_at: datetime
    updated_at: datetime
    volk_id: str
    session_id: str
    relevance_score: float | None


class MemorySummary:
    content: str
    token_count: int
    source_count: int
    source_keys: list[str]
    strategy: SummarizationStrategy
    created_at: datetime


class SummarizationStrategy(Enum):
    RELEVANCE = "relevance"      # Rank by semantic similarity to current context
    RECENCY = "recency"          # Rank by temporal proximity
    IMPORTANCE = "importance"    # Rank by explicit importance scores
Neo4j Memory Schema
CREATE CONSTRAINT memory_key IF NOT EXISTS
FOR (m:Memory) REQUIRE m.key IS UNIQUE;

CREATE INDEX memory_volk IF NOT EXISTS
FOR (m:Memory) ON (m.volk_id);

CREATE INDEX memory_session IF NOT EXISTS
FOR (m:Memory) ON (m.session_id);

CREATE INDEX memory_created IF NOT EXISTS
FOR (m:Memory) ON (m.created_at);

-- (:Memory)-[:BELONGS_TO]->(:Session)
-- (:Memory)-[:CREATED_BY]->(:Volk)
-- (:Memory)-[:RELATES_TO]->(:Task)
-- (:Memory)-[:PART_OF]->(:Project)
-- (:Memory)-[:SUMMARIZED_BY]->(:Memory)
-- (:Memory)-[:SHARED_WITH]->(:Volk)
-- (:Memory)-[:DERIVED_FROM]->(:Memory)
Design Notes
Integration Points
Component Integration
orchestrata-core IMemoryEngine interface, ComponentType.MEMORY_ENGINE
orchestrata-kernel Context injection before Volk execution
orchestrata-session Hot-to-cold migration on session end
orchestrata-knowledge Cold storage in same Neo4j instance
orchestrata-cxp Memory references in CXP packet metadata dict
orchestrata-governance PAM++ memory access policy enforcement
orchestrata-agents Volk memory access during execution
Kernel Context Injection

The Kernel injects memory context into Volk execution at the existing governance check point in runtime.py, after Volk definition lookup and before execution begins.

Injection flow:

  1. Kernel calls memory_engine.search(query=task_context, scope=MemoryScope(volk_id=volk.alias, session_id=current_session))
  2. Results ranked by relevance + recency
  3. Kernel calls memory_engine.summarize(context_id=session_id, max_tokens=memory_budget) to fit within token budget
  4. Summary injected into Volk system prompt via metadata["memory_context"] on the CXP packet

Configuration:

# configs/production.yaml
runtime:
  memory_engine:
    injection_budget_tokens: 4096    # max tokens for memory context
    injection_strategy: "relevance"  # relevance | recency | hybrid
    injection_enabled: true          # toggle memory injection
Session Lifecycle Integration

Memory migration is triggered by the existing on_session_close hook in session/hook_registry.py.

Migration flow:

  1. on_session_close fires with session_id and outcome
  2. Hook calls memory_engine.migrate_to_cold(session_id)
  3. Migration reads all hot entries from Redis (orchestrata:memory:{session_id}:*)
  4. Each entry persisted as (:Memory) node in Neo4j with relationships
  5. Redis keys deleted after successful Neo4j write (atomic per-entry)
  6. Migration count logged as session fact via FactEmitter

Failure handling:

Async bridge: The Memory Engine subscribes to session.closed events via the Kernel EventBus during initialize(). When a session closes, the Kernel emits a session.closed event, and the EventBus delivers it asynchronously to the Memory Engine's handler, which calls migrate_to_cold(session_id). This eliminates the sync/async mismatch entirely — no changes to SPEC-006 required, no direct Redis pub/sub needed.

Prolog Governance Predicates

The Memory Engine extends the PAM++ rule base with these Prolog predicates:

%% Memory access governance
can_store_memory(VolkId, Classification) :-
    registered(VolkId),
    memory_classification_allowed(VolkId, Classification).

can_read_memory(VolkId, MemoryKey) :-
    registered(VolkId),
    (memory_owner(MemoryKey, VolkId) ;
     memory_classification(MemoryKey, shared) ;
     memory_classification(MemoryKey, system)).

can_share_memory(VolkId, TargetVolkId, MemoryKey) :-
    memory_owner(MemoryKey, VolkId),
    registered(TargetVolkId),
    \+ memory_share_blocked(VolkId, TargetVolkId).

can_delete_memory(VolkId, MemoryKey) :-
    memory_owner(MemoryKey, VolkId).

%% Default: all volks can store private memories
memory_classification_allowed(_, private).
%% Shared/system require explicit grant
memory_classification_allowed(VolkId, shared) :- has_role(VolkId, team_lead).
memory_classification_allowed(VolkId, system) :- has_role(VolkId, system_admin).
CXP Integration

Memory references use the existing metadata dict on CXP packets. No schema change to SPEC-004 required.

Convention:

packet.metadata["memory_context"] = {
    "injected_keys": ["mem-uuid-1", "mem-uuid-2"],
    "token_budget": 4096,
    "tokens_used": 3812,
    "strategy": "relevance",
    "summary": "..."  # compressed summary text
}
Event Delivery

All memory events are emitted through the Kernel's EventBus (see SPEC-001 EventBus addendum). The Memory Engine does not push to Redis directly — it calls kernel.emit_event().

Emission pattern:

await kernel.emit_event(SystemEvent(
    type="memory.store.completed",
    source="SYS:MEMORY",
    payload={"key": entry.key, "volk_id": volk_id, "classification": classification},
    severity="info",
))

Topic mapping:

Event Topic Severity
Memory stored memory.store.completed info
Memory retrieved memory.retrieve.completed info
Search completed memory.search.completed info
Summarization done memory.summarize.completed info
Migration completed memory.migrate.completed info
Migration failed memory.migrate.failed error
Token budget warning memory.budget.warning warning
Access denied memory.access.denied warning

Reactive subscriptions (registered during initialize()):

# Subscribe to session close → trigger hot-to-cold migration
await kernel.event_bus.subscribe(
    "session.closed",
    self._on_session_closed,
    subscriber_id="SYS:MEMORY",
)

This replaces the Redis pub/sub bridge described in Session Lifecycle Integration — the EventBus handles the async delivery, eliminating the sync/async hook mismatch.

Redis Hot Storage Format

Key patterns:

orchestrata:memory:{session_id}:{volk_id}:{key}     — HASH (entry data)
orchestrata:memory:{session_id}:{volk_id}:index      — ZSET (keys sorted by created_at)
orchestrata:memory:{session_id}:{volk_id}:tags:{tag}  — SET  (keys with this tag)
orchestrata:memory:embed:{key}                        — STRING (JSON-serialized float[] vector)
orchestrata:memory:embed:index:{session_id}           — ZSET (keys sorted by relevance cache)

TTL: Matches session TTL (default 86400s / 24h). Extended to 604800s (7 days) on migration failure.

Hash fields for entry:

content     — str (raw memory content)
summary     — str | null
tags        — JSON list[str]
metadata    — JSON dict
classification — "private" | "shared" | "system"
volk_id     — str
session_id  — str
task_id     — str | null
created_at  — ISO 8601
updated_at  — ISO 8601
token_count — int
Embedding Configuration

Embedding model is configurable via configs/production.yaml and swappable without code changes.

Configuration:

runtime:
  memory_engine:
    embedding:
      model: "all-MiniLM-L6-v2"          # sentence-transformers model name
      dimension: 384                       # output vector dimension
      cache_embeddings: true               # cache computed embeddings in Redis
      batch_size: 32                       # batch size for bulk embedding
      device: "cpu"                        # cpu | cuda

Similarity search:

Token Counting

Token counting uses a provider-aware abstraction, shared with SPEC-011 Provider Manager.

Interface:

class ITokenCounter(ABC):
    @abstractmethod
    def count(self, text: str) -> int: ...

    @abstractmethod
    def truncate(self, text: str, max_tokens: int) -> str: ...

Implementations:

The active counter is determined by the provider selected for the current Volk (from SPEC-011 Provider Manager). Falls back to WordCounter when provider is unknown.

Success Criteria

  1. Interface Complete: IMemoryEngine implemented with all 6 operations (store, retrieve, search, summarize, delete, migrate_to_cold)
  2. Dual Storage: Redis hot + Neo4j cold working with automatic migration
  3. Search Quality: Semantic search returns relevant results with > 0.8 precision@10
  4. Performance: Hot operations < 50ms, cold operations < 200ms (P95)
  5. Governance: PAM++ enforced on all memory operations
  6. Context Fit: Summarization respects token budgets within 5% tolerance
  7. Integration: DI-registered, Kernel-injectable, CXP-referenceable

Dependencies

Revision History

Date Version Author Changes
2026-03-09 1.0 Mohamed Initial specification