Operational KG migration — Neo4j → orchgraph

Session 254. Plan of record: plans/neo4j-to-orchgraph-migration-plan.md in the orchestrata-ultimate repo (6 phases). Engine comparison and the measurements that justified the move: Graph Engine Analysis.

Why

Neo4j Community structurally cannot cluster — HA is Enterprise-only, closed source, no self-build path. The operational KG is the system's record of fact (sessions, threads, CXP packets, decisions, specs, harness history, code intelligence) and it was running on a single un-replicated instance. The orchgraph fork already provides live Raft HA, MAGE (140 procedures), vector search, and in-memory latency the AST/code-intelligence SLOs need.

Ratified decisions

Two graphs, still not one: memgraph (the default database) remains the shared semantic/RAG graph that both the wiki indexer and orchestrata's semantic_graph write to. orchkg is the operational KG. Same cluster, isolated storage, separate database= on every session — the two-graph doctrine survives the move.

The root-connectivity invariant (ratified this session)

Every node in the operational KG must have a path to the root (:Project) node. Garbage is deleted, never linked.

User, verbatim: "the node must eventually lead to the root project node".

Enforced by scripts/check_kg_connectivity.py — a Python BFS from the Project hub, engine-portable (uses id(), no GDS/APOC, no engine-specific procedures), exit 1 if anything is unreachable — wired into scripts/health_check.py.

What it caught, and what each fix was:

The lesson that generalizes: a writer that creates a node without an edge is a bug, not an untidiness. Sweeping is a bandage; the gate finds them, the writer is where you fix them.

Execution record

Phase 0 — inventory + dialect gate (DONE)

• scripts/gen_kg_migration_inventory.py → plans/kg-migration-inventory.json: 58 files touch the graph (42 through the repository seam, 16 raw driver), 164 queries total.
• scripts/kg_dialect_preflight.py runs every inventoried query against both engines: 102/103 PASS. One real dialect failure (kg_context.py:462, ORDER BY on a raw variable after collect()), five admin-dialect differences. Report: reports/kg-dialect-failures-2026-07-26.md.
• The migration is de-risked by that number: the Cypher surface is portable.

Phase 1 — cluster + schema + backup (DONE)

• orchkg database created on the existing cluster; 147 DDL statements applied via scripts/kg_schema.py apply --dialect memgraph (71 constraints + 71 key indexes + 5 extra). Second pass is a no-op. Live: 71 constraints, 76 indexes.
• Memgraph DDL dialect differs from Neo4j: CREATE CONSTRAINT ON (n:L) ASSERT n.p IS UNIQUE, CREATE INDEX ON :L(p), no IF NOT EXISTS, and constraints do not create backing indexes — every key needs an explicit index or the sub-ms SLOs do not hold.
• Backup: scripts/backup_orchgraph.sh + user units orchgraph-backup.{service,timer} (daily 03:00, Persistent=true, bolt-port readiness guard). Resolves MAIN from SHOW INSTANCES each run — the role moves on failover. Issues CREATE SNAPSHOT per logical database; a multi-tenant snapshot covers only its own database. Writes a manifest.json ({database: node_count}) because Memgraph does not expose the name → storage-uuid mapping over Bolt.
• Restore drill executed: a scratch container built from the orchkg snapshot file alone came up with 71 constraints + 76 indexes matching source.

Phase 2 — access-layer port (DONE)

• A database= parameter is threaded through the whole operational-graph seam (both core/graph adapters + interfaces, core/neo4j_factory, knowledge_graph config/client/context) behind the KG_* env seam. Unset = read-through to the Neo4j defaults, so nothing changes until KG_ENGINE=memgraph is set. The test suite maps each worker onto its own orchkg_w<N> database under that flag.

• core/graph/retry.py: bounded jittered retry for MVCC write conflicts. constraint violation is deliberately in the marker list — measured live, a concurrent MERGE race against a uniqueness constraint can commit-fail as a ClientError, not a TransientError. Retrying is still correct: the loser re-runs MERGE and sees the winner's row.

• Two real dialect bugs surfaced by the first true Memgraph run, both fixed portably: ORDER BY resolving against the post-RETURN scope when the match variable shadows the return alias, and raw verification sessions opened without database= landing on the cluster default while the write went to orchkg_w0.

• Dual-fleet proof: the same 8-item batch is 208 passed / 0 failed on BOTH engines. Six named modules, back-to-back, all faster on the new fleet:

  Module	Neo4j	orchkg	Δ
  test_kg_sync_edges	5.45s	2.62s	−52%
  test_cold_store_expanded	2.81s	1.50s	−47%
  test_sync	2.81s	1.95s	−31%
  test_volk_kg_sync	2.48s	1.77s	−29%
  test_component_work_context	2.28s	1.80s	−21%
  handlers/test_knowledge	19.87s	18.45s	−7%

Phase 3.1 — the migration tool (BUILT)

scripts/migrate_kg_to_orchgraph.py: dry-run by default / --apply, batched 500, MERGE-only on canonical merge keys, resumable via a per-label progress file, and a --checksum mode comparing per-label node counts, per-triple edge counts, and a deterministic per-label property-hash sample.

• Property-type fidelity was verified against the live cluster, not assumed: temporal values and lists round-trip unchanged; bytes/bytearray fail outright on Memgraph and are converted to base64 under an explicit, counted rule. No silent coercion.
• Live dry run: 37,469 nodes / 68,909 relationships would migrate, excluding 11 derived labels (30,342 nodes) that Step 3.3 re-derives from source. Predicate/Rule are in that exclusion set because code_parser.py parses them out of the Prolog sources — derived by the same property that excludes CodeFunction. The 16 live Rule edges that consequently do not copy are recorded as a re-derivation obligation for the cutover check.

Phase 3.1 — the migration RUN (DONE, session 255)

• 42,243 nodes / 71,892 relationships migrated, zero dropped endpoints.
• The tool over-reported success at first: _write_rel_batch returned the number of rows SENT, not written, so 71,842 was claimed against 71,826 in the graph. Fixed with RETURN count(r) plus a counted dropped_endpoints block that prints loudly. A migration that cannot fail visibly has not been verified.

Phase 3.2 — query-parity harness (DONE)

scripts/kg_query_parity.py: 31 record-of-fact queries across root / session / thread / RTM / audit / volk / context / harness / browser, plus opt-in derived queries. Both graphs must answer identically.

• Rows are normalised before comparison — engines return different concrete temporal classes and row orders for the same stored fact; a value change is a failure, a representation difference is not.
• Whole-graph censuses carry the excluded-label filter, and a test fails any future unfiltered census, so "correct by design" diffs cannot train the reader to ignore diffs.
• Parity caught a real loss: the U:Qa Volk was missing from live Neo4j (source 16 vs target 17). Restored through the production boot path (PipelineRunner.boot() → sync_volks_to_kg), not by hand-writing a node.

Phase 3.3 — re-derive the AST subgraph (DONE)

Both graphs re-derived from the same plans/ast-ultimate.json. The derived subgraph now matches exactly: 6/6 labels, 6/6 edge types — a full-fidelity proof of the ported sync layer against real data.

Equality came from deleting garbage in Neo4j, not adding data to orchgraph:

• 41 stranded derived nodes (_test_satn_*, kg_*, pytest tmpdirs) sat as hub-unreachable islands, inflating Neo4j to 787 Modules / 1498 Classes / 805 Packages. A subtree only becomes hub-reachable once its Project->Package edge exists, so a run that died between the node write and the edge write left an island that project-scoped cleanup could not see.
• The Rule re-derivation obligation recorded in Phase 3.1 was superseded, not performed. Ontology ruling 3 retired the Rule label in favour of Constraint{kind:"invariant"}. Live Neo4j carried BOTH shapes — 6 Constraints AND 6 Rules, 10 MUST_FOLLOW->Constraint AND 10 MUST_FOLLOW->Rule — the retired one as duplicate residue, not data of record. orchgraph already held the canonical shape at identical counts. Re-deriving Rule would have imported retired residue into the graph that replaces Neo4j. The harness now asserts the canonical shape for parity and the retired shape as an absence on the target.

Phase 3.4 — live code intelligence (DONE)

All five ratified SLOs met, measured against the live cluster:

scripts/benchmark_code_intel.py + scripts/ast_live_sync.py + orchestrata-ast-live-sync.service (enabled, tracked in deploy/systemd/). Live-proven: saving a real module put its new function in the graph within 12 s of the write.

Every defect here was found by RUNNING the thing:

• Method.id had no index. Method is outside canonical_schema.py's canon, so it gets no CONSTRAINT — and therefore none of the implicit indexes every other code label receives. A canonical-id lookup planned ScanAll over 5,561 nodes. This is why each latency SLO also EXPLAINs its query and fails on a scan: a latency number alone cannot distinguish "fast because indexed" from "fast because the graph is small today."
• A p99 over 200 samples is noise. One SLO "missed" at 3.135 ms p99 with a 13.884 ms single-sample max while its p50 matched its peers exactly; at n=1000 it measured 0.720 ms. Default raised to 1000.
• Analyzer root is load-bearing. Module ids are dotted paths relative to the root, so parsing src/ instead of src/orchestrata_ultimate renames every module and an "incremental" sync ADDS a second codebase: one pass took orchkg from 783 Modules to 1621. load_or_analyze treats a root mismatch as a cache miss and silently full-reparses — fine for a CLI, catastrophic for a service — so the service now refuses to run on a mismatch.
• touch() measures nothing: freshness is content-hashed, so the first SLO run reported PASS at 0.552 s having written 0 nodes.
• Deletions were reported but never applied. A deleted file has no AST entry left, so incremental_sync cannot express the operation at all — the sync logged "1 removed" while the module's whole subtree stayed in the graph.

Phase 4 — cutover (2026-07-26)

• Freeze: orchestrata-kg-retention.timer + orchestrata-history-mining.timer stopped. Final delta re-run (MERGE), then .env flipped, then a SECOND delta to catch the flip window — cheaper and safer than trying to stop the per-turn conversation-capture hook, which follows .env on its next fire.
• .env: NEO4J_URI → an orchgraph coordinator, KG_ENGINE=memgraph, KG_DATABASE=orchkg. NEO4J_USERNAME/PASSWORD retained for rollback and simply ignored — orchgraph's auth module is off (verified: basic_auth, empty credentials, and no auth all connect).
• The cutover was itself a landmine. tests/conftest.py resolved the suite's graph as KG_ROUTING_URI or MEMGRAPH_URI — harmless only while that key was blank. The cutover FILLS it with a production coordinator, and production carries orchkg_w0..w7 exactly like the dedicated test node does, so the flip would have routed every test worker into a per-worker database on the production cluster, writing test data beside the real orchkg, with nothing failing to signal it. TEST_MEMGRAPH_URI now takes precedence, with a regression test that reads the .env FILE so it cannot be made vacuously true by the same env rewrite it checks.
• gen_level0.py learned the post-cutover topology: after the flip NEO4J_URI names a Memgraph coordinator, so a Level-0 block derived from it alone would print "Neo4j" for a Memgraph cluster and keep describing "two graphs" when there is one cluster with two databases — introducing exactly the confusion that document exists to prevent. Detected from the addresses, not restated in prose, so the doc cannot disagree with .env.

Operational notes worth keeping

• MVCC, not locks. Memgraph aborts a transaction on write conflict instead of blocking. Every writer needs a retry wrapper; code ported from Neo4j that assumed pessimistic locking will surface conflicts under parallelism.
• Never poll a Memgraph instance with count/read queries while a bulk edge write runs against it — storage.access_timeout aborts the writer. Watch the writer's stdout, not the database.
• Scratch databases leak. Two mvcc_probe_test* databases left by port testing were dropped; the backup manifest lists every database each run, which makes stragglers visible without a separate audit.