Orchestrata Ultimate › Plans › briefs › 2026-09-22-adr-0058-ha-sandbox-fixture

Brief: retarget the HA controller tests at the two-node orchgraph sandbox

Source: plans/briefs/2026-09-22-adr-0058-ha-sandbox-fixture.md at 8a2cb42ad · last changed 2026-09-22 · first committed 2026-09-22 · 135 lines.

Prose pending.

Plan text

Brief: retarget the HA controller tests at the two-node orchgraph sandbox

DESIGN_WAIVER: ADR-0058 (Accepted 2026-09-22). Scope is exactly the ADR's execution.

GOAL

tests/semantic_graph/test_memgraph_ha.py never touches the shared test orchgraph (10.53.2.10) again. Its register / demote / failover tests run against the two real sandbox nodes and assert real outcomes. Every test in the module passes, in any order, twice in a row, and neither sandbox node logs an assertion.

FACTS (measured 2026-09-22, do not re-derive)

• Sandbox nodes, project orchestrata-test, same orchgraph fork build as the shared node, empty data dirs, systemd unit orchgraph.service:
  • memgraph-ha-main 10.53.2.20:7687, replication port 10000
  • memgraph-ha-replica 10.53.2.21:7687, replication port 10000
• Smoke-tested through MemgraphHAController's own methods: _set_role to replica, _register_replica -> True, _show_replicas reports {'memgraph': {'status': 'ready', 'behind': 0}}, promote replica + demote main + re-register old main -> both True, restore (DROP all replicas, both MAIN) clean. Zero assertion lines in either journal.
• The crash mechanism the ADR records: registering an UNREACHABLE replica on a node and then running SHOW REPLICAS aborts the Debug-build fork. Therefore: no test may register an unreachable endpoint on a real node. Unreachable endpoints (bolt://127.0.0.1:1) stay allowed only where the unreachable node is the one being contacted (set-role / check-node / register-against-unreachable-main tests).
• Profile loading: env/test.env keys land in os.environ verbatim (profile overrides .env). The module's memgraph_config fixture currently reads MEMGRAPH_URI, which is the shared node. Remove that coupling from this module.

FILES

• env/test.env: add, next to TEST_MEMGRAPH_URI, a commented block:

  # --- HA controller sandbox (ADR-0058) ---
  # Two disposable orchgraph nodes the HA tests MAY mutate (roles, replica
  # registration). The shared node above is never a replication target.
  # Containers memgraph-ha-main / memgraph-ha-replica in orchestrata-test.
  # Blank = the HA tests fail loud with their own message, never skip.
  TEST_MEMGRAPH_HA_MAIN_URI=bolt://10.53.2.20:7687
  TEST_MEMGRAPH_HA_REPLICA_URI=bolt://10.53.2.21:7687
  

• env/dev.env: same two keys, blank, with a one-line note (dev has no sandbox yet). Do not touch env/prod.env.
• tests/semantic_graph/test_memgraph_ha.py: fixture + test changes below.
• No changes to src/. If a controller defect surfaces, STOP and report.

DESIGN (settled; implement as written)

1. Delete the module-local memgraph_config fixture.
2. Add a function-scoped ha_sandbox fixture:
   • reads TEST_MEMGRAPH_HA_MAIN_URI / TEST_MEMGRAPH_HA_REPLICA_URI; blank or unset -> pytest.fail("ADR-0058: TEST_MEMGRAPH_HA_*_URI not set in the active env profile; the HA tests need the two-node sandbox"). Never skip.
   • builds main = NodeConfig(name="memgraph-db", bolt_uri=..., replication_host=<host>) and replica = NodeConfig(name="memgraph-replica", bolt_uri=..., replication_host=<host>). Keep the name memgraph-db so the single-node assertions stay valid.
   • a _reset(nodes) helper, run in setup AND teardown, that for each node: if role is main, DROP REPLICA every name from SHOW REPLICAS; then SET REPLICATION ROLE TO MAIN when not already main. Use orchestrata_ultimate.core.graph_factory.create_graph(uri, auth=NO_AUTH) with an auto-commit session (replication commands refuse explicit transactions). Teardown runs even when the test failed.
   • yields (main, replica).
3. ha_controller fixture now depends on ha_sandbox and builds the controller with nodes=[main] (single node, sandbox MAIN). All existing single-node tests keep working unchanged.
4. New ha_pair_controller fixture: same but nodes=[main, replica].
5. Rewrite these tests on ha_pair_controller with REAL expectations:
   • test_reconcile_roles_when_main_is_healthy: mark both healthy, main role main, replica role unknown; _reconcile_roles(); assert persisted main is memgraph-db and _show_replicas(main) lists memgraph-replica with every db status ready (poll up to 10 s).
   • test_ensure_replicas_registered_attempts_registration -> rename to test_ensure_replicas_registered_registers_real_replica: same registration assertion; second call is a no-op (registration left alone: call _ensure_replicas_registered twice, still one entry).
   • test_demote_stray_mains_attempts_demotion -> ..._demotes_real_stray_main: both nodes MAIN and healthy, replica state role main; call _demote_stray_mains(main_state); assert _query_role(replica) == "replica" and it is registered on main.
   • test_perform_failover_falls_back_to_main_or_unknown_candidates: persisted main memgraph-db role main healthy, replica role unknown healthy; _perform_failover(); assert persisted main is memgraph-db (the main/unknown fallback picked it) and the replica ended registered.
   • test_perform_failover_promotes_real_node -> promotes the REAL replica: persisted main memgraph-db marked unhealthy, replica healthy role replica; _perform_failover(); assert persisted main is memgraph-replica, epoch bumped, _query_role(replica) == "main".
   • test_reconcile_roles_triggers_failover_when_main_unhealthy: same shape via _reconcile_roles().
   • test_perform_failover_attempts_promotion keeps its unreachable candidate (it never contacts the real node) — leave as is.
6. Module docstring: state the two-node sandbox, ADR-0058, and the rule "the shared test node is never a replication target".
7. TestNoteReplicaProgress is pure; untouched.

CONSTRAINTS

• No mocks, fakes, stubs, monkeypatching. No pytest.skip. No xfail.
• Read Python through the AST toolkit (scripts/serialize_codebase_ast.py source|symbols|refs), never grep/cat/Read on .py.
• Test runs: background only, observed, timeout 300, with -p orchestrata_ultimate.core.threading.pytest_live_observer, e.g. ORCHESTRATA_ENV_PROFILE=test timeout 300 uv run python -m pytest tests/semantic_graph/test_memgraph_ha.py -p orchestrata_ultimate.core.threading.pytest_live_observer -q -rA with run_in_background: true, then read the output file.
• Never name 10.53.2.10 in this module or in any command.
• Commit with git -c user.name=Mohamed -c user.email=msamy@sufigeeks.org, no AI attribution lines. Do not push.

DONE WHEN

• The module passes twice consecutively (second run proves teardown restores baseline), 0 failed / 0 skipped / 0 xfailed.
• ruff check and ruff format --check clean on the file.
• Both sandbox journals show zero Assertion lines during the runs: incus exec memgraph-ha-main --project orchestrata-test -- journalctl -u orchgraph.service --since -15min --no-pager | grep -c Assertion (and the replica).
• After the runs, both sandbox nodes are MAIN with no replicas (check via MemgraphHAController._query_role / _show_replicas).
• One commit on orchestrata-ultimate master containing env/test.env, env/dev.env, the test module, and this brief unchanged.

VERIFY (report these verbatim)

• The pytest summary lines of both runs.
• The two grep -c Assertion counts.
• The restored roles/replicas of both nodes.
• The commit sha and git show --stat of it.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks