Orchestrata Ultimate › Plans › semantic-graph-ontology-design

Semantic-Graph Ontology Design

Source: plans/semantic-graph-ontology-design.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-08 · 182 lines.

Prose pending.

Plan text

Semantic-Graph Ontology Design

Status: decisions RESOLVED (2026-07-08) — ready to build · Author: orchestrator session Task: #3 (ontology-first; import Neo4j through the ontology, never raw) Anchors: SPEC-008 (KG), SPEC-106 FR-001 (canonical_schema.py), plans/kg-schema-gaps.md. Non-goal: SPEC-105 domaingen/ (OWL/SHACL/Stardog) is a separate system — do not conflate.

1. Why this exists

We are NOT dumping Neo4j into Memgraph. Raw import copies the mess — duplicate label trees (Class+CodeClass, Function+CodeFunction, Spec+Specification), Session written 3 different ways (257k dupes), and 450k+ rows of operational churn — producing the same "numbers make no sense" problem in a second graph. This doc defines the target ontology every node/edge must conform to, and the mapping that cleans Neo4j on the way in.

The ontology is also the fix for Memgraph's current mess (everything flattened to :Entity, four chunk labels): both are symptoms of ingesting without a governing vocabulary.

2. Current state — three vocabularies + a messy graph

Live Neo4j carries 48 labels — the schema.py set plus undeclared synonyms and churn (see §4). Live Memgraph (.192, shared with knowledge-wiki) is Entity-flattened with a 4-way chunk-label split. The ontology must reconcile all of this into ONE canon.

3. Target ontology

3.1 Design principles

1. One canonical label per concept. Synonyms converge (CodeClass→Class). No parallel trees.
2. One merge key per label (extends CANONICAL_MERGE_KEYS), enforced by a uniqueness constraint. Import MERGEs on it → dedup is structural, not a cleanup step.
3. Semantic layer, not an operational log. High-volume append-only telemetry (GatewayAuditEvent, CXPPacket, per-run Timeout/Event) is not semantic knowledge and is excluded from the semantic graph (it stays queryable in Neo4j).
4. Methods are Function nodes with a class_id property (per FR-001) — not a separate Method label.
5. Namespace by project. Every node carries project (orchestrata-ultimate | knowledge-wiki) so the shared cluster stays queryable per-project without separate instances.
6. Chunks reference entities, don't replace them. RAG Chunks (one label: SemanticChunk) attach to canonical nodes via DESCRIBES, so retrieval and structure share one graph.

3.2 Canonical node types (proposed full set)

Tier A — Code structure (from AST sync): Module, Package, Class, Function (methods = Function+class_id), File, Directory, Parameter, Predicate, Rule.

Tier B — Specs & requirements: Spec, Requirement, RequirementItem, UserStory, Scenario.

Tier C — Runtime & provenance: Volk, Session, Project, Portfolio, Intent, PipelineRun, Task, WorkItem, Component, Decision, Artifact, Action.

Tier D — Testing: Test (converges TestNode/DLTest), TestRun, TestResult.

Tier E — NEW / semantic-graph-specific:

• Tool — first-class node for the registry (the original ask). Sourced from tools.registry.all_metadata() (name, category, description, requires_approval), NOT from Neo4j (Neo4j only has ToolCall events). Merge key: name.
• SemanticChunk — the single canonical embedding-chunk label (unifies SemanticChunk/CodestralChunk/QwenChunk/Chunk). Merge key: id. Carries embedding
  • embedder property so multiple embedders coexist under ONE label + ONE vector index.

Tier F — Design (own layer, per decision D-DESIGN-LAYER): DesignModule, DesignClass, Architecture, Pattern. Kept distinct from Tier A code-structure (these are intended design, not the as-built AST). Linked to structure via IMPLEMENTS/REALIZES and to specs via TRACES_TO. Merge keys: DesignModule/DesignClass→id, Architecture→id, Pattern→name.

Tier G — Knowledge & memory (in the semantic graph, per decision D-MEMORY): Fact (governance/derived knowledge, SPEC-010) and Memory (agent memory entries). Both are first-class semantic nodes so retrieval spans code + specs + provenance + accumulated knowledge. Merge keys: Fact→id, Memory→id. Memory carries volk/session scope properties (kept, not dropped) so agent-scoping survives.

3.3 Relationship taxonomy (canonical)

• Containment/structure: DEFINES (Module→Class/Function; Class→Function), CONTAINS, HAS_FILE, CALLS, INHERITS, IMPORTS, DEPENDS_ON, OVERRIDES.
  • Decision #6: HAS_FUNCTION (spec name) vs HAS_METHOD (impl) → add HAS_FUNCTION as the canon, keep HAS_METHOD as a deprecated alias during migration.
• Spec linkage: HAS_SPEC, DEFINES_REQUIREMENT, HAS_ITEM, TRACES_TO (→RequirementItem), IMPLEMENTS, TESTED_BY.
• Provenance: MADE (Volk→Decision), APPLIES_TO, SUPERSEDES, JUSTIFIED_BY, GENERATED_BY, DERIVED_FROM, EXECUTED_BY, TRIGGERED, HAS_SESSION, WORKED_ON(+_BY), ASSIGNED_TO, OWNS/OWNED_BY, PART_OF, REQUIRES.
  • Decision #2: these ~10 provenance edges are declared but unwritten — the ontology defines them; wiring the writers (+ populating Component) is a follow-on build.
• Semantic: DESCRIBES (SemanticChunk→any canonical node), USES_TOOL (Volk/PipelineRun→Tool).

3.4 Merge keys (extends CANONICAL_MERGE_KEYS)

Inherit FR-001's 13. Add: Tool→name, Task→id, Component→name, File→path, Package/Directory→path, Test→id, TestRun→id, UserStory→id, Portfolio→name, Action→id, SemanticChunk→id, DesignModule/DesignClass/Architecture→id, Pattern→name, Fact→id, Memory→id. Each gets a uniqueness constraint via the existing apply() DDL path. (WorkItem/Run/Agent are converged away, so they get no key of their own.)

Key reconciliations (resolved against source-of-truth at build time, correcting this doc's first-draft guesses): Portfolio→name (SPEC-008 contract graph-schema.cql portfolio_name_unique is authoritative — draft said slug). Component→name (all live nodes + the live constraint use name; the contract's path never had conforming data — draft said id; the cql contract's component_path_unique is flagged for reconciliation). Task→id (reuses the live task_id constraint on Task.id — draft said task_id).

4. Neo4j → ontology mapping (the cleaning rules)

adopt = keep as-is · converge = merge into canonical label · exclude = not semantic (stays in Neo4j) · add = create from another source · decide = open question.

Net: ~746k raw Neo4j nodes → a semantic graph on the order of ~25–30k meaningful, deduped, typed nodes + the registry Tools + existing chunks under one label.

5. Decisions — RESOLVED (2026-07-08, user)

1. D-EXCLUDE (churn): ✅ Exclude GatewayAuditEvent/CXPPacket/audit/telemetry from the semantic graph — they stay in Neo4j (operational, 450k rows).
2. D-DESIGN-LAYER: ✅ Own layer — DesignModule/DesignClass/Architecture/Pattern form Tier F, distinct from as-built code structure.
3. D-MEMORY: ✅ Both in the semantic graph — include Fact AND Memory (Tier G). (Memory keeps its volk/session scope properties so agent-scoping survives.)
4. D-AGENT/RUN/WORKITEM: ✅ Converge all three — Agent→Volk, Run→PipelineRun, WorkItem→Task (synonyms of canonical types).
5. D-ENUMS (gap #4): ✅ Retire types_enums.py NodeType/RelationshipType; the canon in canonical_schema.py is the single vocabulary source.
6. D-SESSION-KEY (gap #7): ✅ Yes — adopt Session→session_id uniqueness constraint; flip the test that asserts its absence.
7. D-VOLK-KEY (gap #8): ✅ Yes — converge Volk on alias everywhere.
8. D-TARGET-HA: ✅ Import into the Memgraph HA cluster MAIN (via the .153 controller), after replication is repaired (MAIN currently has 0 registered replicas — prerequisite).

All ontology decisions are settled. Remaining open items are build-time only: the exact write-sites for the unwritten provenance edges (§3.3 Decision #2) — a follow-on, not a blocker.

6. Importer contract (built only after the ontology is ratified)

• Reads Neo4j read-only (Neo4j is untouched — user directive).
• For each mapped source label: stream nodes, project properties onto the canonical shape, MERGE (n:<CanonLabel> {<mergekey>: ...}) SET n += props, n.project = 'orchestrata-ultimate'.
• Converge synonyms by mapping to the same canonical label + key (dedup falls out of MERGE).
• Import only relationships whose both endpoints were imported; map rel type via the taxonomy.
• Add Tool nodes from tools.registry; add USES_TOOL from ToolCall aggregation.
• Idempotent (re-runnable), resumable, and gated by a conformance check (extend canonical_schema.check() to the full label set) that fails on any non-canonical node.

7. How this reconciles the three vocabularies

• Extend canonical_schema.py to the full label/key/constraint set (the REQ-106-008 "later wave" the module docstring defers) — it becomes the single source of truth.
• Reduce schema.py LABEL_*/REL_* to re-exports of the canon (no second definition).
• Retire types_enums.py per D-ENUMS.
• Add the completeness test (gap #9) asserting the exact node/rel/key sets — pins the target so it can't silently drift.

8. Sequencing

1. Ratify §3 + resolve §5 decisions.
2. Extend canonical_schema.py + completeness test (gaps #1,#3,#6,#9 — no-decision builds).
3. Repair cluster replication (prerequisite for a durable import).
4. Build the importer (§6) + conformance gate.
5. Follow-on: wire the unwritten provenance writers (#2), migrate the 6 writers onto the canon.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks