Home / Infrastructure / Memgraph HA cluster build log (historical)

Memgraph HA Cluster — Complete Build Log (historical)

Current state, checked 2026-10-23: none of the containers on this page exists in incus list --all-projects. The live cluster is orchgraph: Memgraph HA cluster reference. Every container on the host is listed on Incus fleets.

DECOMMISSIONED 2026-07-10. This page documents the old Memgraph cluster — a 3-node Memgraph Community cluster (memgraph-1/2/3) fronted by a custom Python HTTP HA controller (memgraph-ha), plus a standalone memgraph-db. That entire cluster was deleted on 2026-07-10 after its data was ported (with a verified parity check) to the current cluster, orchgraph. The container hostnames and IPs below no longer exist. It is retained only as a historical record of the design and rationale.

For the live cluster, see Memgraph HA Cluster Reference (orchgraph). orchgraph is a custom C++ fork of Memgraph Community 3.11 with Raft-based HA (3 coordinators + 2 data instances); it has no HTTP controller — failover is coordinator-managed, and clients connect with a routing driver at neo4j://10.53.1.160:7691.

This page describes how the (now-deleted) Memgraph cluster was built and why the decisions were made.

Why Memgraph

We needed a scalable, performant graph database for:

• RAG and GraphRAG over the knowledge wiki
• Semantic graph storage
• Codebase AST and dependency analysis

We evaluated Neo4j, TigerGraph, KùzuDB, and Memgraph. Memgraph was chosen because:

• Native Cypher support (Neo4j-driver compatible)
• Better single-node performance for our workloads
• Simpler operational model than causal clustering
• A fork of the open-source core is acceptable for our use case

What We Built

A 3-node Memgraph data cluster with a custom Python HA controller. (All containers/IPs below were deleted 2026-07-10 — historical.)

Historical topology (deleted 2026-07-10):

The current orchgraph layout that replaced this: 3 Raft coordinators (orchgraph-coord-1/2/3, bolt 7691) + 2 data instances (orchgraph-data-1/2, bolt 7687), client entry neo4j://10.53.1.160:7691. See the orchgraph reference.

Current live topology — orchgraph (Raft HA)

0. Prerequisites

• Incus installed and a project configured (default is fine).
• A Ubuntu Noble image available: incus launch images:ubuntu/noble ...
• Network access to download.memgraph.com.
• The orchestrata-ultimate source tree at ~/workspace/orchestrata-ultimate-architecture/.

1. Launch the Three Memgraph Data Nodes

Run the node launcher once per node. It is idempotent — if a container exists it is started instead of recreated.

cd ~/workspace/orchestrata-ultimate-architecture/incus

./memgraph-node-launch.sh memgraph-1
./memgraph-node-launch.sh memgraph-2
./memgraph-node-launch.sh memgraph-3

What happens inside each container

1. apt-get update and install dependencies:

apt-get update -qq
apt-get install -y -qq wget curl ca-certificates libssl3 libcurl4 libicu74

2. Download and install Memgraph 3.11.0:

cd /tmp
wget -q https://download.memgraph.com/memgraph/v3.11.0/ubuntu-24.04/memgraph_3.11.0-1_amd64.deb
dpkg -i memgraph_3.11.0-1_amd64.deb || apt-get install -y -f -qq
rm -f memgraph_3.11.0-1_amd64.deb

3. Write /etc/memgraph/memgraph.conf:

--data-directory=/var/lib/memgraph/mg_data
--bolt-address=0.0.0.0
--bolt-port=7687
--monitoring-address=0.0.0.0
--monitoring-port=7444
--metrics-address=0.0.0.0
--metrics-port=9091
--log-level=INFO
--storage-properties-on-edges=true
--query-modules-directory=/usr/lib/memgraph/query_modules

4. Create a systemd override so Memgraph loads its bundled libstdc++:

mkdir -p /etc/systemd/system/memgraph.service.d
cat > /etc/systemd/system/memgraph.service.d/ld-library-path.conf <<EOF
[Service]
Environment=LD_LIBRARY_PATH=/usr/lib/memgraph
EOF

5. Enable and start the service:

systemctl daemon-reload
systemctl restart memgraph
systemctl enable memgraph

6. Wait for port 7687 to listen.

At this point each node is a standalone Memgraph instance. They are not yet clustered.

2. Launch the HA Controller

The controller turns three standalone instances into a replication cluster.

./memgraph-cluster-launch.sh

What the script does

1. Calls memgraph-node-launch.sh for memgraph-1, memgraph-2, memgraph-3.
2. Collects each container's Incus IP.
3. Builds MEMGRAPH_HA_NODES:

memgraph-1|bolt://<node1-ip>:7687|<node1-ip>,memgraph-2|bolt://<node2-ip>:7687|<node2-ip>,memgraph-3|bolt://<node3-ip>:7687|<node3-ip>

4. Creates the memgraph-ha controller container from images:ubuntu/noble.
5. Installs Python, pip, venv, sqlite3.
6. Creates /opt/memgraph-ha/venv.
7. Installs uv, then uses it to install:

fastapi uvicorn[standard] neo4j anyio httpx

8. Copies controller source into the container:

/opt/memgraph-ha/src/orchestrata_ultimate/__init__.py
/opt/memgraph-ha/src/orchestrata_ultimate/semantic_graph/

9. Writes /etc/systemd/system/memgraph-ha.service:

[Unit]
Description=Memgraph HA Controller
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/memgraph-ha
Environment=MEMGRAPH_HA_NODES=memgraph-1|bolt://<node1-ip>:7687|<node1-ip>,...
Environment=MEMGRAPH_HA_PORT=8080
Environment=MEMGRAPH_HA_HOST=0.0.0.0
Environment=MEMGRAPH_HA_DB_PATH=/var/lib/memgraph-ha/state.db
Environment=MEMGRAPH_HA_REPLICATION_MODE=ASYNC
Environment=MEMGRAPH_HA_POLL_INTERVAL=2.0
Environment=PYTHONPATH=/opt/memgraph-ha/src
ExecStart=/opt/memgraph-ha/venv/bin/python -m orchestrata_ultimate.semantic_graph.ha_controller_cli
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

10. Reloads systemd, enables and starts the service.
11. Waits for port 8080.

3. How the Controller Bootstraps the Cluster

On first start:

1. Parses MEMGRAPH_HA_NODES into NodeConfig objects.
2. Opens SQLite at MEMGRAPH_HA_DB_PATH.
3. Creates the cluster_state table if it does not exist.
4. Loads persisted current_main and epoch from SQLite.
5. If no state exists, bootstraps with the first node in the list as MAIN.
6. Saves state to SQLite.

4. How Replication Is Established

During each poll cycle:

1. The controller queries every node for its current role.
2. If the persisted MAIN is healthy, it ensures that node is MAIN and the others are registered as REPLICAs.
3. If a node is down, it is marked unhealthy.
4. If the MAIN is down, a failover is triggered.

Cypher run on the MAIN:

SET REPLICATION ROLE TO MAIN;
REGISTER REPLICA `memgraph-1` ASYNC TO "<node1-ip>:10000";
REGISTER REPLICA `memgraph-3` ASYNC TO "<node3-ip>:10000";

Cypher run on each replica:

SET REPLICATION ROLE TO REPLICA WITH PORT 10000;

5. Autofailover Sequence

When the current MAIN stops responding:

1. Controller marks it unhealthy.
2. Picks the next healthy node from the node list.
3. Runs SET REPLICATION ROLE TO MAIN on the promoted node.
4. Registers remaining healthy nodes as REPLICAs against the new MAIN.
5. Increments epoch.
6. Saves new state to SQLite.
7. /main now returns the new MAIN URI.

6. Autofailback Sequence

When the original MAIN recovers:

1. Controller detects it is healthy.
2. Demotes the temporary MAIN back to REPLICA.
3. Re-promotes the original preferred node to MAIN.
4. Re-registers all nodes against the recovered MAIN.
5. Increments epoch.
6. Saves state.

This was the primary requirement — the reason we built custom HA instead of using Enterprise.

MAIN-role lifecycle across a failover/failback cycle:

7. Verify the Cluster

curl http://<memgraph-ha-ip>:8080/main
curl http://<memgraph-ha-ip>:8080/status

From inside a data node:

incus exec memgraph-2 -- mgconsole -u "" -p "" --host <node2-ip> --port 7687
SHOW REPLICAS;

8. Configure Clients

Set in ~/workspace/knowledge-wiki/.env:

MEMGRAPH_HA_CONTROLLER_URL=http://<memgraph-ha-ip>:8080
MEMGRAPH_USERNAME=
MEMGRAPH_PASSWORD=

The repository client calls /main and connects to the returned Bolt URI.

9. Day-2 Operations

Restart a replica:

incus restart memgraph-1
# Controller re-detects and re-registers it automatically.

Force failover:

curl -X POST http://<memgraph-ha-ip>:8080/failover

Force failback:

curl -X POST 'http://<memgraph-ha-ip>:8080/failback?preferred=bolt://<node1-ip>:7687'

Deploy fresh cluster:

cd ~/workspace/orchestrata-ultimate-architecture/incus
./memgraph-cluster-launch.sh

10. Client Integration

Repository client: ~/workspace/knowledge-wiki/graph-indexer/memgraph_client.py

It:

• Discovers MAIN via the HA controller /main endpoint
• Connects with the Neo4j driver
• Provides typed upserts for Page, Project, Concept, Agent, Service, Procedure, Decision, Blocker, Session, Tag
• Manages relationships and vector indexes

Usage example:

from graph_indexer.memgraph_client import get_connection

client = get_connection()
client.upsert_project(slug="orchestrata-ultimate", title="Orchestrata Ultimate")
client.upsert_service(name="memgraph-cluster", description="HA Memgraph cluster")
client.close()

11. RAG / GraphRAG Integration

The knowledge wiki uses Memgraph for the structured graph and a local sentence-transformers vector store for semantic search. The RAG query pipeline can expand results via graph traversal:

wiki-rag-query "Explain the repository pattern" --graph-expand --graph-depth 1 --graph-limit 10

This finds the top vector matches, then traverses LINKS_TO/MENTIONS relationships in Memgraph to add related pages to the context.

12. Replication Details

• Default mode: ASYNC.
• One MAIN, two REPLICAs.
• Controller registers replicas and re-registers them after restarts.
• Failover and failback are driven by the Python HA controller container.

13. Vector Search

The current Memgraph build has the vector_search module available. We keep the local sentence-transformers store as the primary vector index to avoid an Enterprise dependency, but we also create a native vector index on Chunk(embedding) for future use.

14. Known Limitations

• The HA controller is a single container. If it dies, the cluster continues to run but automatic failover stops until the controller restarts.
• A future iteration is a 3-node Raft controller cluster.
• Split-brain protection relies on the controller's single view.

Related

• Memgraph HA cluster reference (orchgraph, live)
• Incus fleets
• Repository Pattern
• Semantic Graph
• HA Controller
• Graph Databases