Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0058

ADR-0058: HA controller tests run against a dedicated two-node orchgraph sandbox, never the shared test node

← ADR-0057 · ADR-0059 →

The record

Date: 2026-09-22 · Status: Accepted (ruled by Mohamed 2026-09-22)

Context

tests/semantic_graph/test_memgraph_ha.py exercises MemgraphHAController against memgraph-db, which resolves through TEST_MEMGRAPH_URI to the shared test-fleet node (container memgraph, project orchestrata-test, 10.53.2.10). Six of its tests mark an unreachable placeholder node (bolt://127.0.0.1:1, replication 127.0.0.1:10000) as healthy=True and drive _ensure_replicas_registered, _demote_stray_mains and _perform_failover against the shared node. That issues DROP REPLICA + REGISTER REPLICA ... TO '127.0.0.1:10000' on a live multi-database node.

Measured 2026-09-22 (full detail in reports/ha-tests-crash-test-orchgraph-2026-09-22.md):

• The node core-dumped twice (09:41:33 and 11:20:07 UTC) on Assertion failed ... replication_handler.cpp at line 449, Expression: 'ok'. The test node runs the orchgraph fork as a Debug build; upstream ReplicationHandler::ShowReplicas() carries DMG_ASSERT(ok) for a registered replica that lacks a storage client on some database, which is exactly the state a registration to an unreachable target leaves behind.
• The crash instant (11:20:07.004) is the SHOW REPLICAS issued by test_ensure_replicas_registered_skips_unhealthy, immediately after test_reconcile_roles_when_main_is_healthy ran the DROP+REGISTER.
• Recovery takes ~4 minutes; the next four tests fail in ~4 ms with connection refused. These are the 5 red tests in kimi's last full run.
• The registration is durable. After the restart the node still reports replica fake with status invalid on four databases and logs Couldn't connect to remote address 127.0.0.1:10000 once a second (600 lines per 10 minutes). Every new run starts from a poisoned node.
• env/test.env is not involved: the same 5 failures reproduce with the coordinator URIs blanked.

The module docstring says the tests run "against the existing single Memgraph container"; test_set_role_to_replica_on_unreachable_node states it must not mutate the shared instance. The register/demote/failover tests violate that intent, and the repo forbids replacing the node with a double.

Decision (proposed)

Give the HA controller tests their own real infrastructure: two disposable orchgraph containers in the orchestrata-test project (copies of memgraph with empty data directories and their own IPs), exposed to the suite as TEST_MEMGRAPH_HA_MAIN_URI / TEST_MEMGRAPH_HA_REPLICA_URI in env/test.env. The ha_controller fixture builds two NodeConfigs from those keys; tests that today register a placeholder on the shared node register the real replica instead and assert real outcomes (registration succeeds, failover promotes the replica, demotion re-registers the old main). Fixture teardown drops every replica and restores both nodes to MAIN so the tests are order-independent. Tests that exercise unreachable behaviour keep their unreachable endpoints but never name the shared node as MAIN. The shared test node is repaired once (DROP REPLICA fake) and is never again a target of replication mutation.

Alternatives rejected

1. Repair only (DROP REPLICA fake on 10.53.2.10). Removes today's poison and the log spam, but the next HA run recreates it and crashes the node again. Necessary, not sufficient.
2. Weaken the tests (mark the placeholder unhealthy so no registration happens, or turn the register/demote/failover tests into unreachable-vs-unreachable checks). Loses the only real-server coverage of those branches; the test-writer rule forbids weakening a test that caught a real defect.
3. Fix the fork (replace DMG_ASSERT(ok) with the Release-build skip and redeploy the test binary). Correct and still wanted, but the fork source is not on this host (/home/mohamed/workspace/orchgraph is absent; the deployed binary's build path names it) and a rebuild does not remove the durable placeholder registration or the shared-node mutation. Tracked as follow-up, not the gate fix.
4. Point the tests at the scratch HA cluster (rh-data-1). That node is coordinator-managed; manual replication commands on it are refused or confuse the coordinators, and kimi's main-discovery tests now depend on that cluster being untouched.

Consequences

• Shipping orchestrata-ultimate needs a green full-suite record; with this decision the 5 HA tests stop crashing the shared node and the gate can run end to end. Until it is executed, ship remains blocked.
• Two more containers in orchestrata-test (memory footprint of an empty orchgraph each) and two env keys whose blank value must fail loud, in the pattern TEST_MEMGRAPH_COORDINATOR_URIS already follows.
• Replication state on the shared test node becomes read-only for the suite; a later guard can assert SHOW REPLICAS is empty there.
• The fork's Debug assertion remains latent until alternative 3 lands.

Provenance

Raised while taking over kimi's in-flight migration work on Mohamed's instruction ("check kimi's files and take over", 2026-09-22). Evidence: reports/ha-tests-crash-test-orchgraph-2026-09-22.md (commit a9554e88), node journal of orchgraph.service on memgraph/orchestrata-test, reports/test-phase-durations.ndjson, upstream src/replication_handler/replication_handler.cpp (ShowReplicas). The auto-mode classifier denied both the node repair and container creation as shared-resource changes, so nothing had been executed at proposal time. Recommendation was: accept this ADR, then repair the node and build the sandbox in that order.

Ruling 2026-09-22 (Mohamed, in-session, one-decision prompt): selected "Accept: sandbox + repair" over "Repair only", "Fix the fork first" and "Reject: rewrite tests without a second node". Execution order as recommended: repair the shared node once, then build the two-node sandbox and retarget the fixture.

Related ADRs

No other ADR is named in this record.

Rulings cited

None.

Specs cited

None (or no spec page exists for the ones named).