Graph Engine Analysis — Neo4j vs orchgraph

Session 254 analysis (2026-07-25/26). Every number here was measured against the live systems, not quoted from vendor docs. Generic engine background lives at Graph Databases; this page is the orchestrata-ultimate-specific ground truth.

The two engines as deployed

Neither engine dominates both roles. Neo4j's structural deficit is closed-source HA; orchgraph's is the RAM-bound dataset plus fork maintenance ownership.

MAGE on orchgraph — why it was missing, how it was deployed

Why it was missing: never built. mage/ has its own build system that build.sh and the release packaging never invoked; the Jul-10 fleet deploy hand-copied exactly one module (vector_search.so) — a deliberate minimal scope, not a failure.

Build (host, CachyOS):

sudo pacman -S --needed boost        # leiden needs boost headers
cd ~/workspace/orchgraph/mage
python3 setup build --lang Cpp Python --type Release --no-ml
# → mage/dist: 36 compiled .so + 22 python modules (6.7 MB)

TRAP: mage/setup does not propagate ninja failures, and piping through tail eats the exit code. Verify .so files exist in mage/dist; never trust the exit code.

Deploy (replica first — HA fails over automatically):

cd mage/dist && tar -cf - . | incus exec orchgraph-data-2 -- tar -C /opt/memgraph/query_modules -xf -
incus exec orchgraph-data-2 -- pacman -S --needed --noconfirm python-networkx python-igraph python-scipy
incus exec orchgraph-data-2 -- systemctl restart orchgraph-data.service
# wait for health=up in SHOW INSTANCES (vector-index rebuild takes minutes), repeat for data-1

Verified on the live graph: pagerank.get() and degree_centrality.get() across all 765K nodes; leiden found 95 communities over 11,145 Entity nodes on a connected projection. 10 python modules (torch/sklearn/elasticsearch/duckdb backed) fail to load by design under --no-ml; memgraph logs and skips them.

Packaging: release tarballs now include MAGE automatically whenever mage/dist exists — orchgraph commit fdf6dd8 (scripts/orchgraph-package.sh + docs/INSTALL.md). Verified by a real packaging run: 197 query-module entries, binary byte-identical to build/memgraph.

Algorithm gotchas: whole-graph leiden_community_detection.get() throws "No communities detected" (most nodes are isolated: 765K nodes, 497K edges) — run it on a connected projection (MATCH p=(a:Entity)-[]-(b:Entity) WITH project(p) AS g CALL leiden_community_detection.get(g) …). Memgraph has no string slicing [..n] — use substring().

Memory (host: 31 GiB, CachyOS)

Fair comparison — the graphs are near-identical in size.

• The in-memory engine holds a comparable graph in ~27 % less RSS than the JVM — Neo4j's figure is mostly heap reservation, not data.
• orchgraph fleet total ≈ 1.62 GiB process RSS buys 2 data copies + HA; Neo4j's 1.07 GiB buys one unreplicated instance — whose RAM does not grow with data (could hold 10× the graph in the same footprint).
• Memgraph RSS grows linearly ≈ 1 GiB per million nodes per copy at current density → practical ceiling ~10–20 M nodes/copy on this host.
• Host-wide top consumers: Neo4j fleet 3.06 GiB (biggest single process is the idle test instance neo4j-node-3 at 1.33 GiB — tarball install pre-commits 1 g heap; nodes 2/4 idle at ~310 MiB with grow-on-demand heap. Tuning opportunity: drop heap.initial_size on test workers). Container plumbing (9 × systemd/journald/etc.) ≈ 1.2 GiB. zram swap: 5.1 GiB cold pages compressed to 1.2 GiB physical (zstd 4.2:1) — healthy, not pressure.

Incidents found & fixed during this analysis

1. neo4j-node-3 dead — its tmpfs mount (opt-neo4j-data.mount) failed at boot with 237/KEYRING (unprivileged-container keyring race), leaving the enabled neo4j.service dead. Remounted + started + verified bolt on 10.53.1.14. Likely one of the boot-health FAIL units. If it recurs, add a retry drop-in. (neo4j-node-4 is fine — different layout, tmpfs at /var/lib/neo4j/data.)
2. session/kg_context.py multi_hop() raised on every call — primary path called apoc.path.subgraphAll but the server has no APOC; _run doesn't swallow driver errors so the if not rows "fallback" was unreachable; the fallback also ignored depth and dropped the start node. Replaced with plain variable-length Cypher (*0..N, int-coerced interpolated depth — Cypher forbids params in var-length bounds). 62 tests pass; live-verified. orchestrata commit 0b248bee.
3. /opt/wiki-js stray content — 15 stale 2026-07-06 page snapshots (from when disk-storage pointed at the install dir) verified file-by-file against the repo (repo newer or identical in every case; concepts/test.md was an orphan of a deleted test page) and deleted. assets/ there is the Wiki.js client bundle — never touch.

Migration analysis: Neo4j → orchgraph, what we would lose

Real losses (structural):

1. RAM-flat growth — the KG is append-only and must hold 100+ projects (multi-project invariant); Neo4j serves it from disk in a fixed footprint, Memgraph needs ~1 GiB RAM per million nodes per replica. The fork's RocksDB on-disk mode (ADR-003) exists but is its least mature path.
2. Forgiving write concurrency — Neo4j blocks, Memgraph aborts + retries; every KG write path (kernel sync, audit, runtime_writer, swarm coordinator) would need retry logic. Permanent tax, already bitten once on the semantic graph.
3. Upstream-maintained storage for the system of record — orchgraph bugs are ours (e.g. the vector-index recovery fix at fork HEAD). Acceptable risk for the semantic layer; doubling the blast radius for the KG is not.

Myths (verified false): GDS (zero call sites; MAGE covers algorithms now), APOC (sole call site was already broken; now portable), uniqueness constraints (supported), the Python neo4j driver (all 29 import sites already work against Memgraph bolt).

Migration cost (one-time, distinct from loss): Cypher dialect audit of all call sites (SHOW INDEXES/dbms.*/string-slicing/datetime differ), fulltext → Tantivy port (1 call site), rework of per-worker test instances + neo4j_test_preflight + pollution sweeps + bootstrap. Weeks, given the verify-everything test culture. Level-0 rule stands: the two graphs must not co-mingle — migration would need a second cluster or strict label isolation.

Verdict: keep the split. Neo4j = unbounded write-heavy system of record; orchgraph = RAM-sized read-heavy vector-native semantic layer. If KG HA is the itch, a Neo4j Community warm-standby (store copy + WAL shipping) is far cheaper than an engine migration.

Open-source alternatives survey (C-based, memory-light)

Filter: genuinely open source (BSL/SSPL/BUSL rejected per standing rule), disk-based (= RAM-light), C/C++.

Either Postgres flavor would idle ~300–500 MB for the current KG and stay flat. Strongest candidate if the KG ever leaves Neo4j: Apache AGE. Zero-migration option: drop heap.initial_size to 512m (~600 MB total) — one config line.

Wiki.js replication runbook (asked in the same session)

Wiki.js 2.5.314 lives in /opt/wiki-js (systemd wiki-js.service, port 3000) with SQLite at /opt/wiki-js/data/db.sqlite — the DB is the source of truth (users, settings, history, navigation); the git repo (gitlab.com/MSamy/knowledge-wiki) is only the disk-storage content mirror. To replicate on another machine: stop service → tar czf /opt/wiki-js → unpack on target (Node ≥18) → clone the repo to the disk-storage path (path is stored in the DB: Admin → Storage, or UPDATE storage SET config=json_set(config,'$.path',…) WHERE key='disk') → copy the unit file → systemctl enable --now wiki-js. Repo alone is NOT enough.

Orphaned-node investigation & purge (2026-07-26, post-page addendum)

Question: after the session-226 test-pollution cleanup, are there orphaned nodes? Answer: the primary was sweeper-clean (0 _test_-marker ids, 0 /tmp paths — the 226 cleanup held) but held 24,331 isolated nodes in three classes; workers held fresh pollution.

Bonus: two leaked CliApplyProbe_test_* uniqueness constraints on the primary (node sweeps can't remove DDL) — dropped by the same tool.

Post-purge verification: all 4 DBs sweeper-clean; primary 824,283 → 813,925 nodes (exactly −10,358); canonical data intact (9,721 connected Parameters). Remaining isolated total 13,973 = Class 3 + the 19 unproven Files.

The great KG cleanup (2026-07-26, second addendum) — 824,283 → 70,840 nodes

The orphan investigation escalated into a full audit-and-purge of the operational KG. 91.4% of the graph was exhaust; every deletion class was user-ratified, dry-run-verified, preserved-then-purged, and turned into repeatable tooling. All four DBs (primary + 3 workers) now report sweeper-clean including deep fingerprints.

Sessions resolved: the 260,082 Session nodes were three concepts sharing one label (gateway connections / CLI work sessions / test husks). After the purges: 180 real sessions, all linked Project-[:HAS_SESSION]-> , 0 isolated (user directive fulfilled).

Retention deployed: systemd --user timer orchestrata-kg-retention (Sun 04:00, --gateway-audit-retention-days 30 --cxp-exhaust --apply) so neither exhaust class can re-accumulate.

CXP communication analysis (docs/cxp-communication-analysis-2026-07-26.md): the 46% clarification bounce is partly mandated (Concierge forced-questions rule); 24% of real clarification rounds answered EMPTY intents; ~75% real bounce rate; 5 ranked intake fixes with exact code targets (reject empty intents in intent_validator, stop re-asking defaulted D2/D6, structured intake fields via unused Intent.resource_path, give U:Cx read_file/ list_dir, make the forced first bounce conditional). Flow analysis: hop latency p50 47ms (machines) vs multi-day tail (humans); bug found: 87 threads dead-end at scope_stories — U:PM→U:Ar dispatch never fires. Dead-letter verdict: delivery_status is written BEFORE dispatch (postman/sync_router.py:82) and never updated — the field carries zero information; 100% of real pending intents were actually consumed.

ADR-0037 ratified: idempotent KG event ids are SHA1 content digests — recovered from the orphaned U:Ar volk Decision node, first volk-originated ADR (7b65334f).

Thread ontology (Option A, user-ratified) — APPLIED: Opus-designed contract (plans/thread-ontology-design.md), Sonnet-built (scripts/backfill_thread_links.py, MERGE-only, dry-run by default, --apply to write). Live result matches the contract exactly: 9,950 Thread nodes, 9,950 HAS_THREAD, 26,107 IN_THREAD, 12,588 BY_VOLK (+202 pre-existing AuditEntry literals now formalized = 12,790 total), 15,603 TO_VOLK. Isolated CXPPacket/Message/ToolCall/Artifact: 0. A second --apply created nothing (idempotency proven by graph recount). Residue: 1,507 distinct unmatched agent strings (23,371 occurrences, overwhelmingly test-<hex>-U:*) — filed as S10 work. Thread uniqueness constraint live on all 4 DBs.

Ops note: an opencode auto-checkpointer was committing mid-session and swallowed two changes into checkpoint: commits (7fb215c8 — sweeper session_id fix + CXP analysis doc; 5729dabe — corpus archive). Content is in history under junk messages; opencode stopped thereafter.

Commits produced by this session

What happened next

This analysis is what the migration decision was made on. The execution — ratified decisions, the multi-tenant orchkg pivot, the root-connectivity invariant, and the phase-by-phase record — lives at Operational KG migration — Neo4j → orchgraph.