Orchestrata Ultimate / Specs / 008-knowledge-graph

008-knowledge-graph: Knowledge Graph

The Neo4j Knowledge Graph is the persistent memory system for Orchestrata's multi-agent coordination. It serves as shared institutional memory that enables agents to maintain context across sessions, understand project relationships, and coordinate effectively without filling agent context windows.

At a glance

Item	Value
Spec directory	`specs/008-knowledge-graph/`
Status line in the spec	Complete — Phase 2 context-injection expansion landed (runtime signals, validator loop, expanded context sections)
Requirement IDs defined	123
With a `@traces` citation in `src/`	101 (82.1%)
With a `@traces` citation in `tests/`	26 (21.1%)
Tracked files	11
RTM page	008-knowledge-graph

Reading this spec

The Neo4j Knowledge Graph is the persistent memory system for multi-agent coordination in Orchestrata. It acts as shared institutional memory: agents keep context across sessions, understand how projects relate, and coordinate without filling their own context windows.

The scope is six user stories: temporal knowledge tracking, portfolio-based organization, the agent memory interface, code analysis integration and knowledge synchronization (all priority 1), and multi-hop retrieval (priority 2). The stated aims are cross-session persistence, relationship understanding, prevention of context window overflow, multi-project portfolio coordination and awareness of how projects evolve over time. Requirement IDs use the REQ-008- prefix.

The design decisions are Neo4j 5.0 or later for storage, Tree-sitter for parsing Python and Prolog code, temporal timestamps on every node and relationship, Cypher query patterns for context injection, and goal-oriented synchronization following the XOS pattern.

In the code, the package is orchestrata_ultimate/knowledge_graph. The largest tracers are orchestrata_ultimate/knowledge_graph/code_parser, orchestrata_ultimate/knowledge_graph/synchronizer, orchestrata_ultimate/knowledge_graph/anomaly_detector, orchestrata_ultimate/knowledge_graph/query_engine and orchestrata_ultimate/knowledge_graph/types. The per-requirement view is on the RTM page.

The spec's status line reads Complete, with a note that a Phase 2 context-injection expansion has landed (runtime signals, a validator loop and expanded context sections). It tracks 11 files and defines 123 requirement IDs. The full text follows below.

Measured coverage is 101 of 123 requirements cited by at least one @traces in src/ (82.1%), spread over 42 modules; the module citing the most of them is orchestrata_ultimate/knowledge_graph/code_parser (21 requirements). In tests/, 26 requirements are cited by a test-side @traces.

Coverage gaps: 22 requirements have no citing symbol. The largest untraced groups are REQ-008-* (22). Untraced means no @traces names the ID; the behaviour may still exist in code that was never annotated. Test-side citations are thinner than code-side ones (26 against 101).

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID	Done	Requirement	Traced by
`REQ-008-001`	yes	Neo4j knowledge graph serves as persistent memory for multi-agent coordination	2
`REQ-008-002`	yes	Knowledge graph maintains temporal awareness with time-stamped nodes and relationships	6
`REQ-008-003`	yes	Portfolio-based organization enables cross-project context sharing	3
`REQ-008-004`	yes	Agents can query knowledge graph for task-relevant context without loading entire project history	6
`REQ-008-005`	yes	Multi-hop retrieval enables complex context gathering across multiple relationships	11
`REQ-008-006`	yes	Knowledge graph supports both static code structure and dynamic execution state	19
`REQ-008-007`	yes	Portfolio nodes can contain multiple projects with shared context	3
`REQ-008-008`	yes	Cross-project dependency tracking enables portfolio-level coordination	1
`REQ-008-009`	yes	Portfolio-level dashboards show progress across multiple projects	2
`REQ-008-010`	yes	Automatic portfolio assignment based on project metadata and tags	none
`REQ-008-011`	yes	Portfolio-level governance extends PAM++ ownership to portfolio scope	none
`REQ-008-012`	yes	Standardized Cypher query patterns for common agent context needs	5
`REQ-008-013`	yes	Context injection protocol provides only relevant nodes to agent context	19
`REQ-008-014`	yes	Query timeout enforcement prevents agent blocking	1
`REQ-008-015`	yes	Caching layer optimizes frequently accessed context	6
`REQ-008-016`	yes	Agent-specific query permissions enforced through PAM++	1
`REQ-008-017`	yes	Tree-sitter integration for Python code analysis	6
`REQ-008-018`	yes	Tree-sitter integration for Prolog code analysis	6
`REQ-008-019`	yes	Incremental parsing enables efficient updates for large codebases	5
`REQ-008-020`	yes	Concrete syntax tree preservation maintains accurate code representation	1
`REQ-008-021`	yes	Code analysis results automatically mapped to knowledge graph nodes	32
`REQ-008-022`	yes	Goal-oriented synchronization following XOS pattern	2
`REQ-008-023`	yes	Idempotent updates ensure consistency across sync operations	17
`REQ-008-024`	yes	Unidirectional flow from project files to knowledge graph	6
`REQ-008-025`	yes	Anomaly correction automatically maintains consistency	12
`REQ-008-026`	yes	File watchers monitor changes for incremental updates	14
`REQ-008-027`	yes	Time-stamped nodes track creation, update, and validity periods	3
`REQ-008-028`	yes	Historical queries enable temporal reasoning and context selection	3
`REQ-008-029`	yes	Versioned entity states support rollback and comparison	7
`REQ-008-030`	yes	Temporal audit capabilities track changes over time	4
`REQ-008-031`	yes	PAM++ ownership enforcement for all knowledge graph queries	4
`REQ-008-032`	yes	Portfolio-level access control for cross-project queries	1
`REQ-008-033`	yes	Agent-specific query permissions based on role and authority	2
`REQ-008-034`	yes	Audit trail maintained for all graph modifications and queries	5
`REQ-008-035`	yes	Governance engine integration enforces memory operation policies	3
`REQ-008-036`	yes	Query response time under 100ms for common context queries	1
`REQ-008-037`	yes	Index optimization for frequent query patterns	4
`REQ-008-038`	yes	Sharding support for large portfolios with thousands of projects	none
`REQ-008-039`	yes	Memory-efficient query patterns prevent resource exhaustion	4
`REQ-008-040`	yes	Distributed query processing for high-concurrency environments	1
`REQ-008-041`	yes	CXP protocol integration for memory context references	3
`REQ-008-042`	yes	PAM++ integration for ownership boundary enforcement	2
`REQ-008-043`	yes	Session management integration for temporal session tracking	4
`REQ-008-044`	yes	Governance engine integration for policy enforcement	3
`REQ-008-045`	yes	Prolog system integration for dynamic state synchronization	3
`REQ-008-046`	yes	Multi-project context injection verification	1
`REQ-008-047`	yes	Temporal query accuracy testing	1
`REQ-008-048`	yes	Portfolio relationship validation	1
`REQ-008-049`	yes	Agent coordination testing across multiple projects	1
`REQ-008-050`	yes	Performance testing under load with large codebases	1
`REQ-008-051`	yes	Comprehensive API documentation for memory interface	none
`REQ-008-052`	yes	Graph schema documentation with node and relationship definitions	4
`REQ-008-053`	yes	Query pattern documentation for common agent scenarios	3
`REQ-008-054`	yes	Integration guide for existing Orchestrata components	none
`REQ-008-055`	yes	Performance tuning guide for large-scale deployments	none
`REQ-008-056`	yes	`CodeParser.resolve_calls` builds a name→function index across all analyzed files and emits a CALLS edge for every call-site match found in a function's `calls[]` list	8
`REQ-008-057`	yes	Resolved CALLS edges are deduplicated per caller→callee pair so a repeated call site never produces a duplicate graph edge	5
`REQ-008-058`	yes	Self-calls (a function calling itself, e.g. recursion) are excluded from CALLS edge resolution	4
`REQ-008-060`	yes	CALLS edges carry a confidence score (`_call_confidence`) that ranks same-file > same-package > global-scope resolution, attenuated when the callee name is ambiguous across multiple candidates	3
`REQ-008-062`	yes	`CodeParser.resolve_inheritance` builds a name→class index across all analyzed files to resolve base-class names to target ClassNodes	4
`REQ-008-063`	yes	Resolved inheritance edges are deduplicated per child→parent pair	3
`REQ-008-064`	yes	Self-inheritance edges are excluded from inheritance resolution	1
`REQ-008-065`	yes	Base-class names are extracted from the AST (`ast.Name` and dotted `ast.Attribute` access) before being handed to inheritance resolution	2
`REQ-008-066`	yes	Base-class names produced via `ast.dump` (unresolved attribute access) are stripped of their dump wrapper before name-index lookup	1
`REQ-008-067`	yes	`CodeParser.resolve_imports` builds a module-path→file mapping across all analyzed files to resolve import statements to target File nodes	5
`REQ-008-068`	yes	Import resolution falls back to partial (prefix) module-path matching when an exact module path is not found in the index	3
`REQ-008-069`	yes	Resolved import edges are deduplicated per source-file→target-file pair	1
`REQ-008-071`	yes	Self-imports (a file resolving to itself) are excluded from import edge resolution	1
`REQ-008-072`	yes	`_extract_class_with_methods`/`ClassExtractor._extract_methods` extract a class's methods as separate FunctionNodes, each carrying signature, parameters, return type, decorators, and docstring	6
`REQ-008-073`	yes	Extracted method FunctionNodes have `class_id` set to their owning ClassNode's id, establishing the Class→Function hierarchy	5
`REQ-008-074`	yes	Method return-type unparsing failures are caught and logged rather than raised, so extraction continues for the remaining methods	2
`REQ-008-075`	yes	`_extract_class_with_methods` returns the ClassNode paired with its method FunctionNodes as a single unit for downstream sync	2
`REQ-008-092`	yes	`ASTAnalyzer.__init__` wires a `CodeParser`, `ClassExtractor`, and `FunctionExtractor` (each defaulting to a fresh instance when not injected) to coordinate file/project analysis	1
`REQ-008-098`	yes	`ASTAnalyzer.analyze_file` parses a single file via `CodeParser.parse_file` and returns its `FileAnalysis`	2
`REQ-008-099`	yes	`ASTAnalyzer.analyze_project` walks a project directory (honoring extension filters and ignore-dir patterns), aggregates per-file analyses into total class/function/predicate/line counts, and records elapsed parse time	2
`REQ-008-100`	yes	`ClassExtractor.extract` returns all top-level class definitions found in a module's AST	1
`REQ-008-101`	yes	`CodeParser._extract_function` extracts a function/method's name, signature, parameters, and metadata from its AST node	1
`REQ-008-059`	yes	`AnomalyDetector.__init__` wires the Neo4j client, `FileSynchronizer`, project id, and an optional asyncio event loop needed for anomaly detection/correction	1
`REQ-008-061`	yes	`AnomalyDetector.connect` ensures the underlying Neo4j client connection is established (idempotent — a no-op when already connected) before detection begins	1
`REQ-008-070`	yes	`AnomalyDetector._get_loop` returns the explicitly-provided event loop when open, otherwise creates/reuses a loop for synchronous correction calls	2
`REQ-008-076`	yes	`_check_single_file` flags a `STALE_FILE` anomaly when a graph File node's path no longer exists on disk, and skips the remaining checks for that file	1
`REQ-008-077`	yes	`_check_hash` flags a `HASH_MISMATCH` anomaly when the on-disk file hash differs from the hash recorded on the graph File node	1
`REQ-008-078`	yes	`_check_line_count` flags a `LINE_COUNT_DRIFT` anomaly when the on-disk line count differs from the graph File node's recorded `line_count`	1
`REQ-008-079`	yes	`_check_entities` re-parses a file's AST and compares class/function names against the graph's recorded entities to detect drift and orphans	2
`REQ-008-080`	yes	`_check_entity_kind` flags an `ORPHAN_RELATIONSHIP` anomaly for each graph entity (class or function) whose name is absent from the current AST	1
`REQ-008-081`	yes	`_find_missing_files` walks the project directory (honoring `SyncConfig` ignore patterns) and flags a `MISSING_FILE` anomaly for every on-disk file absent from the graph	1
`REQ-008-082`	yes	`_correct_stale_file` corrects a `STALE_FILE` anomaly by cascade-deleting the file's nodes from the graph	2
`REQ-008-087`	yes	`_correct_missing_file` corrects a `MISSING_FILE` anomaly by syncing the file into the graph via `FileSynchronizer.sync_file`	1
`REQ-008-088`	yes	`_correct_hash_mismatch` corrects a `HASH_MISMATCH` anomaly by invalidating the cached file hash and re-syncing the file	1
`REQ-008-089`	yes	`_correct_orphan_relationship` corrects an `ORPHAN_RELATIONSHIP` anomaly by deleting the orphaned node from the graph	2
`REQ-008-090`	yes	`_correct_line_count_drift` corrects a `LINE_COUNT_DRIFT` anomaly by updating the graph File node's `line_count` property to match disk	2
`REQ-008-091`	yes	`_apply_corrections` routes each detected anomaly through auto-correction (`mode='auto'`) or per-anomaly user confirmation (`mode='interactive'`), moving uncorrectable or failed anomalies into the report's `uncorrectable` list	2
`REQ-008-083`	yes	`TemporalManager.version_node` closes the previous version (`valid_until = now`) before creating a new version with `valid_from = now` and `valid_until = None`	1
`REQ-008-084`	yes	Each node version is assigned a unique version id and stores a snapshot of the node's properties, independent of later mutations to the caller's dict	2
`REQ-008-085`	yes	`version_node` persists the new property state to Neo4j (`updated_at` + properties) when a connected client is available, and degrades gracefully (logs a warning) when persistence fails	1
`REQ-008-086`	yes	`TemporalManager.get_state_at` returns the node version whose `[valid_from, valid_until)` window contains the requested timestamp, or `None` if no version was valid at that point in time	1
`REQ-008-093`	yes	`SpikeAuditor.log_packet` persists a routed CXP packet as a `(:CXPPacket)` node keyed by `message_id` via an idempotent `MERGE` (safe to log the same packet more than once)	2
`REQ-008-094`	yes	The `(:CXPPacket)` node carries the packet's envelope metadata — `thread_id`, `from_agent`, `to_agent`, `subject`, serialized `payload`, and `timestamp`	1
`REQ-008-095`	yes	The `(:CXPPacket)` node also carries `priority`, `delivery_status`, and `work_status` so the audit trail can be queried for routing/delivery/work state	1
`REQ-008-096`	yes	`log_packet` failures (e.g. Neo4j unavailable) are caught and logged rather than raised, so a missing audit trail never breaks packet routing	3
`REQ-008-097`	yes	`SpikeAuditor.log_routing` records a `[:ROUTED_TO {subject, thread_id}]` relationship between two existing `(:CXPPacket)` nodes by `message_id`, reconstructing the routing path for a thread	2
`REQ-008-102`	yes	`GraphContext.to_prompt` assembles the files/relationships/tests/recent-changes/requirements/decisions sections into a single instructive text block and truncates it to fit `max_context_tokens` before returning it for a volk prompt	11
`REQ-008-103`	yes	`GraphContext.recent_changes` is populated from explicit `(:File)-[:CHANGED_IN]->(:Change)` records, `(:File)-[:NEXT_VERSION]->(:File)` version chains, or file timestamp properties	2
`REQ-008-104`	yes	`GraphContext.similar_patterns` is populated with files whose classes share base classes or implemented protocols with the target files	none
`REQ-008-105`	yes	`GraphContext.semantic_files` is populated by querying the Memgraph semantic-graph chunk store and mapping chunk file paths back to Neo4j `File` roots	none
`REQ-008-106`	yes	`GraphContext.requirements` is populated by resolving `TRACES_TO` links from target symbols to `RequirementItem` nodes and unifying them with parent `Requirement`/`Spec` chains	1
`REQ-008-107`	yes	`GraphContext.decisions` is populated by querying `(:Decision)-[:APPLIES_TO]->(file\|symbol)` relationships	1
`REQ-008-108`	yes	Requirement closure walks `(:Requirement)-[:REQUIRES]->(:Requirement)` edges from traced requirements up to a bounded depth	none
`REQ-008-109`	yes	Requirement closure deduplicates results by canonical requirement id and records shortest hop distance	none
`REQ-008-110`	yes	`RuntimeSignalWriter` persists runtime signals (`Message`, `ToolCall`, `Decision`, `GatewayEvent`) to the Neo4j graph with deterministic ids	none
`REQ-008-111`	yes	Runtime signals are linked to mentioned `File`/`Class`/`Function` nodes via `MENTIONS` edges	none
`REQ-008-112`	yes	Inter-agent messages are written as `Message` nodes carrying `thread_id`, `sender`, `recipient`, `subject`, and `summary`	none
`REQ-008-113`	yes	Tool invocations are written as `ToolCall` nodes carrying `tool_name`, `arguments_json`, `result_summary`, and `outcome`	none
`REQ-008-114`	yes	Governance decisions are written as `Decision` nodes carrying `rationale`, linked to requirements via `JUSTIFIED_BY`, and to volks via `MADE`	none
`REQ-008-115`	yes	Gateway audit events are written as `GatewayEvent` nodes carrying `event_type`, `outcome`, and `summary`	1
`REQ-008-116`	yes	`PipelineKGMixin` exposes a runtime signal writer and persists decisions via `sync_decisions_to_kg`	none
`REQ-008-117`	yes	`BaseAgent` exposes a runtime-signal writer setter and writes tool-call signals	none
`REQ-008-118`	yes	Kernel pipeline dispatch writes `Message` signals for routed CXP packets	none
`REQ-008-119`	yes	Both kernel and orchestrator runners configure agents with the runtime signal writer	none
`REQ-008-120`	yes	`GraphContextValidator` validates an LLM response against the core entities cited in `graph_context`	none
`REQ-008-121`	yes	`ValidatorLoop` retries an LLM call with expanded context when validation reports missing core citations	none
`REQ-008-122`	yes	`ContextExpansion` carries `extra_hops`, `include_tested_by`, `include_requirement_closure`, and `limit_multiplier` controls	1
`REQ-008-123`	yes	Each retry applies a broader expansion while preserving the original payload as the base for every attempt	none

Code that traces this spec

Module	Symbols	Requirements
`orchestrata_ultimate/gateway/handlers/knowledge`	2	2
`orchestrata_ultimate/kernel/pipeline_kg`	16	8
`orchestrata_ultimate/kernel/pipeline_session`	1	1
`orchestrata_ultimate/knowledge_graph/anomaly_detector`	25	16
`orchestrata_ultimate/knowledge_graph/ast_analyzer`	3	3
`orchestrata_ultimate/knowledge_graph/class_extractor`	5	6
`orchestrata_ultimate/knowledge_graph/code_parser`	17	21
`orchestrata_ultimate/knowledge_graph/config`	3	3
`orchestrata_ultimate/knowledge_graph/context_injector`	24	7
`orchestrata_ultimate/knowledge_graph/context_validator_loop`	1	1
`orchestrata_ultimate/knowledge_graph/cxp_auditor`	6	6
`orchestrata_ultimate/knowledge_graph/exceptions`	1	1
`orchestrata_ultimate/knowledge_graph/file_watcher`	10	1
`orchestrata_ultimate/knowledge_graph/function_extractor`	1	1
`orchestrata_ultimate/knowledge_graph/graph_client`	3	3
`orchestrata_ultimate/knowledge_graph/graph_context_validator`	2	2
`orchestrata_ultimate/knowledge_graph/graph_node_mixins`	12	7
`orchestrata_ultimate/knowledge_graph/history`	5	4
`orchestrata_ultimate/knowledge_graph/interfaces`	7	11
`orchestrata_ultimate/knowledge_graph/kg_design`	1	1
`orchestrata_ultimate/knowledge_graph/nodes`	3	1
`orchestrata_ultimate/knowledge_graph/pam_integration`	3	2
`orchestrata_ultimate/knowledge_graph/portfolio`	6	4
`orchestrata_ultimate/knowledge_graph/project_resolver`	1	1
`orchestrata_ultimate/knowledge_graph/prolog_bridge`	7	5
`orchestrata_ultimate/knowledge_graph/query_engine`	13	13
`orchestrata_ultimate/knowledge_graph/relationships`	2	3
`orchestrata_ultimate/knowledge_graph/schema`	2	2
`orchestrata_ultimate/knowledge_graph/search`	1	5
`orchestrata_ultimate/knowledge_graph/sync`	26	10
`orchestrata_ultimate/knowledge_graph/synchronizer`	23	16
`orchestrata_ultimate/knowledge_graph/temporal`	4	9
`orchestrata_ultimate/knowledge_graph/test_results_store`	3	1
`orchestrata_ultimate/knowledge_graph/types`	7	11
`orchestrata_ultimate/knowledge_graph/types_enums`	2	2
`orchestrata_ultimate/knowledge_graph/types_nodes`	6	4
`orchestrata_ultimate/knowledge_graph/versioning`	7	2
`orchestrata_ultimate/orchestrator/pipeline_runner`	1	3
`orchestrata_ultimate/session/kg_sync/__init__`	1	1
`orchestrata_ultimate/session/kg_sync/_base`	1	2
`orchestrata_ultimate/session/kg_sync/_stage_code`	1	1
`orchestrata_ultimate/session/kg_sync/_stage_session`	3	4

Checklists

`checklists/constitutional.md`

Knowledge Graph Constitutional Checklist

Security and Governance Principles

[x] CONST-008-001: Knowledge graph must enforce PAM++ ownership boundaries for all queries and updates
[x] CONST-008-002: All memory operations must be governed by Prolog-based policy rules
[ ] CONST-008-003: Portfolio-level access control must extend PAM++ governance to cross-project operations
[x] CONST-008-004: Agent-specific query permissions must be enforced based on role and authority
[x] CONST-008-005: All graph modifications must be logged to audit trail for compliance

Data Integrity and Consistency

[ ] CONST-008-006: Knowledge graph must maintain data integrity through ACID transactions
[x] CONST-008-007: Synchronization must be idempotent to prevent data corruption
[x] CONST-008-008: Unidirectional flow from project files to knowledge graph must be enforced
[ ] CONST-008-009: Anomaly correction must automatically maintain consistency without manual intervention
[x] CONST-008-010: Temporal data integrity must be maintained with proper time-stamping and versioning

Privacy and Data Protection

[ ] CONST-008-011: Knowledge graph must not store sensitive information without encryption
[ ] CONST-008-012: Personal data must be anonymized or pseudonymized where possible
[ ] CONST-008-013: Data retention policies must be enforced for temporal data
[ ] CONST-008-014: Access to sensitive nodes must require additional authentication
[x] CONST-008-015: Data minimization principles must be applied to context injection

System Reliability and Resilience

[x] CONST-008-016: Knowledge graph must maintain availability during partial system failures
[ ] CONST-008-017: Query timeout enforcement must prevent system resource exhaustion
[ ] CONST-008-018: Graceful degradation must be implemented during high load conditions
[ ] CONST-008-019: Backup and recovery procedures must be defined and tested
[x] CONST-008-020: System must recover gracefully from Neo4j connection failures

Ethical and Responsible AI

[x] CONST-008-021: Knowledge graph must not enable harmful or malicious agent behaviors
[x] CONST-008-022: Agent coordination must be transparent and explainable
[x] CONST-008-023: Decision-making processes must be auditable and traceable
[x] CONST-008-024: System must prevent unauthorized escalation of agent capabilities
[ ] CONST-008-025: Human oversight must be maintained for critical portfolio decisions

Compliance and Regulatory Requirements

[ ] CONST-008-026: System must comply with data protection regulations (GDPR, CCPA, etc.)
[ ] CONST-008-027: Audit trail must meet regulatory requirements for system changes
[ ] CONST-008-028: Access controls must meet industry security standards
[ ] CONST-008-029: System must support compliance reporting and monitoring
[ ] CONST-008-030: Data sovereignty requirements must be respected for international deployments

Performance and Resource Management

[x] CONST-008-031: Query performance must not degrade system responsiveness
[x] CONST-008-032: Memory usage must be optimized to prevent resource exhaustion
[x] CONST-008-033: Index optimization must balance performance and storage requirements
[ ] CONST-008-034: Distributed query processing must not compromise data consistency
[ ] CONST-008-035: Resource allocation must be fair across multiple agents and portfolios

Interoperability and Standards

[ ] CONST-008-036: CXP protocol integration must follow established message format standards
[ ] CONST-008-037: API design must follow RESTful principles and OpenAPI specifications
[x] CONST-008-038: Graph schema must use standard property naming conventions
[ ] CONST-008-039: Event schema must follow established event-driven architecture patterns
[x] CONST-008-040: Integration with existing Orchestrata components must maintain backward compatibility

Transparency and Explainability

[x] CONST-008-041: Knowledge graph queries must be explainable to human users
[x] CONST-008-042: Context injection decisions must be transparent and auditable
[x] CONST-008-043: Portfolio relationships must be clearly visible and understandable
[x] CONST-008-044: Agent coordination logic must be traceable through the knowledge graph
[x] CONST-008-045: System behavior must be predictable and consistent

Innovation and Future-Proofing

[x] CONST-008-046: Knowledge graph schema must be extensible for new node types
[x] CONST-008-047: Query patterns must be adaptable to new agent capabilities
[x] CONST-008-048: Synchronization architecture must support new programming languages
[ ] CONST-008-049: Portfolio management must scale to support enterprise-level deployments
[x] CONST-008-050: System must be designed for continuous improvement and evolution

Human-AI Collaboration

[x] CONST-008-051: Knowledge graph must enhance human understanding of system state
[ ] CONST-008-052: Portfolio dashboards must provide actionable insights for human users
[x] CONST-008-053: Agent coordination must be controllable and modifiable by humans
[ ] CONST-008-054: System must support human-in-the-loop decision making
[x] CONST-008-055: Knowledge graph must facilitate human oversight of AI activities

Environmental and Sustainability Considerations

[x] CONST-008-056: Query optimization must minimize computational resource usage
[x] CONST-008-057: Synchronization processes must be energy-efficient
[ ] CONST-008-058: System must support sustainable scaling practices
[ ] CONST-008-059: Resource allocation must consider environmental impact
[x] CONST-008-060: System design must promote long-term sustainability

Community and Open Source Principles

[x] CONST-008-061: Knowledge graph design must be compatible with open source principles
[ ] CONST-008-062: Documentation must be comprehensive and accessible
[x] CONST-008-063: System must support community contributions and extensions
[x] CONST-008-064: Standards compliance must enable interoperability with other systems
[ ] CONST-008-065: Innovation must be shared with the broader community where appropriate

`checklists/requirements.md`

The requirement list above comes from this file; 123 of 123 checkbox items are ticked.

`checklists/testing.md`

Knowledge Graph Testing Checklist

Functional Testing

[x] TEST-008-001: Verify Neo4j knowledge graph serves as persistent memory for multi-agent coordination
[x] TEST-008-002: Validate temporal awareness with time-stamped nodes and relationships
[x] TEST-008-003: Test portfolio-based organization with cross-project context sharing
[x] TEST-008-004: Verify agents can query knowledge graph for task-relevant context without loading entire project history
[x] TEST-008-005: Validate multi-hop retrieval for complex context gathering across multiple relationships
[x] TEST-008-006: Test knowledge graph support for both static code structure and dynamic execution state

Portfolio Management Testing

[x] TEST-008-007: Verify portfolio nodes can contain multiple projects with shared context
[x] TEST-008-008: Test cross-project dependency tracking for portfolio-level coordination
[x] TEST-008-009: Validate portfolio-level dashboards showing progress across multiple projects
[ ] TEST-008-010: Test automatic portfolio assignment based on project metadata and tags
[ ] TEST-008-011: Verify portfolio-level governance extends PAM++ ownership to portfolio scope

Agent Memory Interface Testing

[x] TEST-008-012: Validate standardized Cypher query patterns for common agent context needs
[x] TEST-008-013: Test context injection protocol provides only relevant nodes to agent context
[ ] TEST-008-014: Verify query timeout enforcement prevents agent blocking
[x] TEST-008-015: Test caching layer optimizes frequently accessed context
[x] TEST-008-016: Validate agent-specific query permissions enforced through PAM++

Code Analysis Integration Testing

[ ] TEST-008-017: Verify Tree-sitter integration for Python code analysis
[ ] TEST-008-018: Test Tree-sitter integration for Prolog code analysis
[ ] TEST-008-019: Validate incremental parsing enables efficient updates for large codebases
[ ] TEST-008-020: Test concrete syntax tree preservation maintains accurate code representation
[x] TEST-008-021: Verify code analysis results automatically mapped to knowledge graph nodes

Synchronization Architecture Testing

[x] TEST-008-022: Validate goal-oriented synchronization following XOS pattern
[x] TEST-008-023: Test idempotent updates ensure consistency across sync operations
[x] TEST-008-024: Verify unidirectional flow from project files to knowledge graph
[ ] TEST-008-025: Test anomaly correction automatically maintains consistency
[ ] TEST-008-026: Validate file watchers monitor changes for incremental updates

Temporal Awareness Testing

[x] TEST-008-027: Verify time-stamped nodes track creation, update, and validity periods
[x] TEST-008-028: Test historical queries enable temporal reasoning and context selection
[x] TEST-008-029: Validate versioned entity states support rollback and comparison
[x] TEST-008-030: Test temporal audit capabilities track changes over time

Security and Governance Testing

[x] TEST-008-031: Verify PAM++ ownership enforcement for all knowledge graph queries
[ ] TEST-008-032: Test portfolio-level access control for cross-project queries
[x] TEST-008-033: Validate agent-specific query permissions based on role and authority
[x] TEST-008-034: Test audit trail maintained for all graph modifications and queries
[ ] TEST-008-035: Verify governance engine integration enforces memory operation policies

Performance and Scalability Testing

[x] TEST-008-036: Validate query response time under 100ms for common context queries
[x] TEST-008-037: Test index optimization for frequent query patterns
[ ] TEST-008-038: Verify sharding support for large portfolios with thousands of projects
[x] TEST-008-039: Test memory-efficient query patterns prevent resource exhaustion
[ ] TEST-008-040: Validate distributed query processing for high-concurrency environments

Integration Testing

[ ] TEST-008-041: Verify CXP protocol integration for memory context references
[x] TEST-008-042: Test PAM++ integration for ownership boundary enforcement
[ ] TEST-008-043: Validate session management integration for temporal session tracking
[ ] TEST-008-044: Test governance engine integration for policy enforcement
[x] TEST-008-045: Verify Prolog system integration for dynamic state synchronization

Multi-Project Testing

[x] TEST-008-046: Test multi-project context injection verification
[x] TEST-008-047: Validate temporal query accuracy testing
[x] TEST-008-048: Test portfolio relationship validation
[ ] TEST-008-049: Verify agent coordination testing across multiple projects
[x] TEST-008-050: Test performance testing under load with large codebases

Edge Case Testing

[x] TEST-008-051: Test knowledge graph behavior with empty portfolios
[x] TEST-008-052: Validate handling of circular dependencies in project relationships
[x] TEST-008-053: Test query performance with deeply nested multi-hop relationships
[ ] TEST-008-054: Verify synchronization behavior with concurrent file modifications
[ ] TEST-008-055: Test governance enforcement with invalid query patterns

Failure Mode Testing

[x] TEST-008-056: Test knowledge graph behavior during Neo4j connection failures
[ ] TEST-008-057: Validate agent behavior during query timeouts
[ ] TEST-008-058: Test synchronization recovery after partial failures
[ ] TEST-008-059: Verify audit trail integrity during system crashes
[ ] TEST-008-060: Test graceful degradation during high memory pressure

Compatibility Testing

[ ] TEST-008-061: Verify compatibility with existing Orchestrata CXP protocol
[x] TEST-008-062: Test integration with existing PAM++ governance system
[ ] TEST-008-063: Validate compatibility with current session management
[ ] TEST-008-064: Test backward compatibility with existing agent implementations
[ ] TEST-008-065: Verify compatibility with different Neo4j versions

Documentation Testing

[ ] TEST-008-066: Validate API documentation accuracy with actual implementation
[x] TEST-008-067: Test graph schema documentation matches actual node structure
[ ] TEST-008-068: Verify query pattern documentation works with real agent scenarios
[ ] TEST-008-069: Test integration guide accuracy with existing Orchestrata components
[ ] TEST-008-070: Validate performance tuning guide recommendations with actual deployments

Other tracked files

specs/008-knowledge-graph/README.md
specs/008-knowledge-graph/compliance-audit.md
specs/008-knowledge-graph/contracts/api.yaml
specs/008-knowledge-graph/contracts/events.yaml
specs/008-knowledge-graph/contracts/graph-schema.cql
specs/008-knowledge-graph/prompt.md
specs/008-knowledge-graph/review-2026-03-25.md

Full spec text

Verbatim from specs/008-knowledge-graph/spec.md; headings are pushed down two levels.

SPEC-008: Knowledge Graph

Metadata

Spec ID: 008-knowledge-graph
Title: Knowledge Graph
Author: Mohamed
Status: Complete — Phase 2 context-injection expansion landed (runtime signals, validator loop, expanded context sections)
Created: 2026-01-15
Last Updated: 2026-07-14
Priority: P1
Components: Neo4j module, Tree-sitter parser, Knowledge Sync
Dependencies: Neo4j 5.0+, Python 3.11+, Tree-sitter

Overview

What

Why

Cross-session memory persistence
Project relationship understanding
Context window overflow prevention
Multi-project portfolio coordination
Temporal awareness of project evolution

How

Neo4j 5.0+ for graph storage
Tree-sitter for code parsing (Python, Prolog)
Temporal timestamps on all nodes/relationships
Cypher query patterns for context injection
Goal-oriented synchronization (XOS pattern)

User Scenarios & Stories

US-001: Temporal Knowledge Tracking (P1)

As an agent I want to access time-stamped project knowledge So that I understand project evolution over time

Acceptance Criteria:

[ ] All nodes have created_at, updated_at, valid_until timestamps
[ ] Historical queries supported via temporal filters
[ ] Versioned entity states trackable
[ ] Temporal reasoning for context selection

US-002: Portfolio-Based Organization (P1)

As a portfolio manager I want to organize projects into portfolios So that I can manage cross-project context

Acceptance Criteria:

[ ] Portfolio nodes contain multiple projects
[ ] Cross-project relationships trackable
[ ] Shared component tracking across projects
[ ] Portfolio-level queries supported

US-003: Agent Memory Interface (P1)

As an agent I want to query the knowledge graph for task context So that I get relevant information without disk reads

Acceptance Criteria:

[ ] Cypher query patterns for common contexts
[ ] Deterministic context injection
[ ] Relevant nodes only (prevent context overflow)
[ ] Multi-hop retrieval for complex relationships
[ ] Query timeout enforcement (< 100ms)

US-004: Code Analysis Integration (P1)

As the system I want to parse code into the knowledge graph So that agents understand code structure

Acceptance Criteria:

[ ] Python code analysis via tree-sitter-python
[ ] Prolog code analysis via custom grammar
[ ] Incremental parsing for efficient updates
[ ] AST preservation for accurate representation
[ ] File → Class → Function → Method hierarchy

US-005: Knowledge Synchronization (P1)

As the system I want to sync project files to the knowledge graph So that graph stays consistent with code

Acceptance Criteria:

[ ] Goal-oriented synchronization (XOS pattern)
[ ] Idempotent updates ensuring consistency
[ ] Unidirectional flow (files → graph)
[ ] Anomaly correction for consistency maintenance
[ ] File watchers for automatic sync

US-006: Multi-Hop Retrieval (P2)

As an agent I want to perform complex multi-hop queries So that I can gather comprehensive context

Acceptance Criteria:

[ ] Query chains across multiple relationship types
[ ] "Find components affected by recent decisions"
[ ] "Show ownership chains for component"
[ ] Variable-depth traversal support

Functional Requirements

Graph Schema

FR-001: Node Types (P1)

System SHALL support 13 core node types:
- Portfolio: {name, description, created_at}
- Project: {name, path, status, created_at}
- Session: {seq, purpose, status, created_at}
- Task: {id, description, status, priority, created_at}
- Component: {name, path, type, language, created_at}
- Volk: {alias, role, model, status, created_at}
- Decision: {rationale, applies_to, created_at}
- Artifact: {filepath, type, created_at}
- File: {path, language, hash, created_at}
- Class: {name, file, line_count, created_at}
- Function: {name, class, file, complexity, created_at}
- Predicate: {name, file, created_at}
- Rule: {name, file, created_at}

FR-002: Relationship Types (P1)

System SHALL support 19 core relationship types:
- (:Portfolio)-[:CONTAINS]->(:Project)
- (:Project)-[:BELONGS_TO]->(:Portfolio)
- (:Project)-[:HAS_SESSION]->(:Session)
- (:Session)-[:CONTAINS]->(:Task)
- (:Session)-[:WORKED_ON_BY]->(:Volk)
- (:Task)-[:REQUIRES]->(:Component)
- (:Task)-[:ASSIGNED_TO]->(:Volk)
- (:Component)-[:OWNED_BY]->(:Volk)
- (:Component)-[:PART_OF]->(:Project)
- (:Decision)-[:APPLIES_TO]->(:Component)
- (:Decision)-[:MADE_BY]->(:Volk)
- (:Volk)-[:OWNS]->(:Component)
- (:Volk)-[:WORKED_ON]->(:Session)
- (:Project)-[:HAS_FILE]->(:File)
- (:File)-[:DEFINES]->(:Class)
- (:File)-[:DEFINES]->(:Function)
- (:Class)-[:HAS_FUNCTION]->(:Function)
- (:File)-[:DEFINES]->(:Predicate)
- (:Rule)-[:DEPENDS_ON]->(:Predicate)

FR-003: Temporal Properties (P1)

System SHALL add created_at timestamp to all nodes
System SHALL add updated_at timestamp to mutable nodes
System SHALL support valid_until for versioned entities
System SHALL index temporal properties for query performance

Context Injection

FR-004: Context Injection Protocol (P1)

System SHALL provide standardized Cypher patterns for context queries
System SHALL inject only relevant nodes into agent context
System SHALL support component ownership queries
System SHALL support recent decision queries
System SHALL support task-in-progress queries

FR-005: Multi-Hop Query Support (P2)

System SHALL support variable-length path queries
System SHALL support relationship type filtering
System SHALL support temporal constraints in path queries
System SHALL limit query depth to prevent resource exhaustion

Code Analysis

FR-006: Python Parsing (P1)

System SHALL extract file, class, function, method nodes
System SHALL extract import relationships
System SHALL extract function call relationships
System SHALL calculate cyclomatic complexity
Implementation note: The live parser uses the Python stdlib ast module rather than tree-sitter-python. The capability set above is unchanged; only the parser dependency differs.

FR-007: Prolog Parsing (P1)

System SHALL extract predicate definitions
System SHALL extract rule definitions
System SHALL extract predicate dependencies
Implementation note: The live parser uses a regex-based Prolog scanner rather than a custom tree-sitter-prolog grammar. The capability set above is unchanged; only the parser dependency differs.

Synchronization

FR-008: File Watching (P1)

System SHALL watch project directories for changes
System SHALL trigger incremental sync on file change
System SHALL handle file creation, modification, deletion
System SHALL debounce rapid file changes

FR-009: Incremental Updates (P1)

System SHALL parse only changed files
System SHALL update only affected graph nodes
System SHALL maintain referential integrity
System SHALL log all sync operations

FR-010: Anomaly Correction (P2)

System SHALL detect inconsistencies between files and graph
System SHALL auto-correct detected anomalies
System SHALL report uncorrectable anomalies
System SHALL support manual reconciliation

Phase 2 Context-Injection Expansion (2026-07-14)

FR-011: Expanded Context Sections (P1)

The context injector SHALL populate recent_changes, similar_patterns, semantic_files, requirements, and decisions sections from the knowledge graph
recent_changes SHALL draw from explicit Change nodes, NEXT_VERSION chains, or file timestamp properties
similar_patterns SHALL surface files whose classes share bases or protocols with target files
semantic_files SHALL bridge the Memgraph semantic-graph chunk store to Neo4j File roots
requirements SHALL resolve TRACES_TO links through RequirementItem to parent Requirement/Spec chains, including REQUIRES closure
decisions SHALL surface Decision nodes that apply to target files or symbols

FR-012: Tier-H Runtime Signal Ingestion (P1)

The graph SHALL persist runtime signals as Tier-H nodes: Message, ToolCall, Decision, GatewayEvent
Each signal SHALL carry a deterministic id and link to mentioned File/Class/Function nodes via MENTIONS
The kernel pipeline, orchestrator runner, and agent runtime SHALL wire the RuntimeSignalWriter

FR-013: Validator Feedback Loop (P2)

A GraphContextValidator SHALL validate LLM responses against core entities cited in graph_context
A ValidatorLoop SHALL retry LLM calls with progressively expanded context on missing citations
Expansion controls SHALL include extra hops, TESTED_BY neighbours, requirement closure, and limit multiplier

Non-Functional Requirements

Performance

NFR-001: Query Performance

Common context queries: < 50ms (P95)
Multi-hop queries (depth 3): < 100ms (P95)
Full-text search: < 200ms (P95)
Concurrent queries: 100+ simultaneous

NFR-002: Synchronization Performance

Single file sync: < 100ms
Batch sync (100 files): < 5 seconds
Incremental sync overhead: < 10% of full sync

NFR-003: Scalability

Support 100,000+ nodes per portfolio
Support 1,000,000+ relationships
Support 100+ concurrent agents querying

Reliability

NFR-004: Data Consistency

ACID transactions for all writes
Eventual consistency for sync operations
Zero data loss during synchronization

NFR-005: Availability

99.9% uptime target
Read replicas for query scaling
Automatic failover support

Security

NFR-006: Access Control

PAM++ ownership enforcement
Portfolio-level access control
Agent-specific query permissions
Query audit logging

Technical Design

Architecture

┌─────────────────────────────────────────────────────────┐
│              Knowledge Graph System                      │
├─────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │   Code      │  │   Context   │  │    Sync     │     │
│  │   Parser    │  │   Injector  │  │   Engine    │     │
│  │ (Tree-sitter)│  │             │  │             │     │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
│         │                │                │             │
│         ▼                ▼                ▼             │
│  ┌─────────────────────────────────────────────────┐   │
│  │              Neo4j Graph Database                │   │
│  │  - Nodes: File, Class, Function, Volk, etc.    │   │
│  │  - Relationships: DEFINES, CALLS, OWNS, etc.   │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Node Schema

# Core Nodes
class PortfolioNode:
    name: str
    description: str
    created_at: datetime

class ProjectNode:
    name: str
    path: str
    status: str
    created_at: datetime

class FileNode:
    path: str
    language: str
    hash: str
    line_count: int
    created_at: datetime

class ClassNode:
    name: str
    file: str
    line_start: int
    line_end: int
    bases: List[str]
    created_at: datetime

class FunctionNode:
    name: str
    class_name: Optional[str]
    file: str
    line_start: int
    line_end: int
    complexity: int
    created_at: datetime

class VolkNode:
    alias: str
    role: str
    model: str
    status: str
    created_at: datetime

Cypher Query Patterns

Component Ownership Query:

MATCH (v:Volk {alias: $volk_alias})-[:OWNS]->(c:Component {name: $component_name})
MATCH (c)<-[:APPLIES_TO]-(d:Decision)
MATCH (c)<-[:REQUIRES]-(t:Task {status: 'in_progress'})
MATCH (s:Session)-[:CONTAINS]->(t)
RETURN c, d, t, s

Recent Decisions Query:

MATCH (v:Volk)-[:MADE]->(d:Decision)-[:APPLIES_TO]->(c:Component)
WHERE d.created_at > datetime() - duration({days: 7})
RETURN v, d, c
ORDER BY d.created_at DESC

AST Subtree Query:

MATCH (f:File {path: $file_path})
OPTIONAL MATCH (f)-[:DEFINES]->(c:Class)
OPTIONAL MATCH (c)-[:HAS_FUNCTION]->(m:Method)
OPTIONAL MATCH (f)-[:DEFINES]->(fn:Function)
OPTIONAL MATCH (f)-[:IMPORTS]->(i:Import)
RETURN f, c, m, fn, i

Integration Points

Component	Integration
orchestrata-cxp	KG context references in packet headers
orchestrata-governance	PAM++ ownership in queries
orchestrata-session	Session persistence as graph
orchestrata-agents	Context injection for Volks

Success Criteria

Schema Complete: All 13 node types and 19 relationship types implemented
Performance: Context queries < 50ms, Multi-hop < 100ms
Coverage: Python and Prolog parsing functional
Sync: File changes reflected in graph within 5 seconds
Scale: Support 100K+ nodes per portfolio

Dependencies

Neo4j 5.0+
Python 3.11+
tree-sitter-python
tree-sitter-prolog (custom)
orchestrata-core (interfaces)
orchestrata-cxp (context headers)

Revision History

Date	Version	Author	Changes
2026-01-15	1.0	Mohamed	Initial specification
2026-03-07	1.1	Mohamed	Standardized format, added numbered requirements

008-knowledge-graph: Knowledge Graph

At a glance

Reading this spec

Requirements

Code that traces this spec

Checklists

checklists/constitutional.md

Knowledge Graph Constitutional Checklist

Security and Governance Principles

Data Integrity and Consistency

Privacy and Data Protection

System Reliability and Resilience

Ethical and Responsible AI

Compliance and Regulatory Requirements

Performance and Resource Management

Interoperability and Standards

Transparency and Explainability

Innovation and Future-Proofing

Human-AI Collaboration

Environmental and Sustainability Considerations

Community and Open Source Principles

checklists/requirements.md

checklists/testing.md

Knowledge Graph Testing Checklist

Functional Testing

Portfolio Management Testing

Agent Memory Interface Testing

Code Analysis Integration Testing

Synchronization Architecture Testing

Temporal Awareness Testing

Security and Governance Testing

Performance and Scalability Testing

Integration Testing

Multi-Project Testing

Edge Case Testing

Failure Mode Testing

Compatibility Testing

Documentation Testing

Other tracked files

Full spec text

SPEC-008: Knowledge Graph

Metadata

Overview

What

Why

How

User Scenarios & Stories

US-001: Temporal Knowledge Tracking (P1)

US-002: Portfolio-Based Organization (P1)

US-003: Agent Memory Interface (P1)

US-004: Code Analysis Integration (P1)

US-005: Knowledge Synchronization (P1)

US-006: Multi-Hop Retrieval (P2)

Functional Requirements

Graph Schema

Context Injection

Code Analysis

Synchronization

Phase 2 Context-Injection Expansion (2026-07-14)

Non-Functional Requirements

Performance

Reliability

Security

Technical Design

Architecture

Node Schema

Cypher Query Patterns

Integration Points

Success Criteria

Dependencies

Revision History

Related

`checklists/constitutional.md`

`checklists/requirements.md`

`checklists/testing.md`