Reproducibility & IaC

How the suite-critical fleet and its two graphs are made reproducible: fleet IP-pinning, an env contract, dynamic Memgraph-MAIN discovery, graph bootstrap tooling, full-fleet Terraform with a drift gate, and CI/local runner parity. Plan of record: plans/reproducibility-plan.md (phases R1–R6).

Status (2026-07-11) — R1–R6 done, both runbook items executed

Phase	What	State
R1	Pin all suite-critical container IPs (fleet ran on DHCP leases)	DONE + restart-tested
R2	`.env.example` (98 keys) + `scripts/validate_env.py` preflight	DONE
R3	`semantic_graph/main_discovery.resolve_main` — never pin the dynamic Memgraph MAIN	DONE
R4	`scripts/bootstrap_graphs.py` (rebuild both graphs from source) + live Neo4j migration	DONE (migration executed)
R5.1/5.2	Full 13-node fleet in Terraform, zero-diff import + drift gate	DONE
R5.3	Rebuild rehearsal from empty	GREEN — empty Incus → app-suite-green, measured
R6	CI + local both run `scripts/run_tests.py`; `validate_ci` enforces it	DONE

R1–R6 all DONE, and the R5.3 rebuild rehearsal is now GREEN end-to-end (see below).

The two graphs — DO NOT CONFLATE

Neo4j 10.53.1.11 = the OPERATIONAL knowledge graph.
Memgraph orchgraph-* (routing 10.53.1.160:7691) = the SHARED semantic/RAG graph, Raft HA (3 coordinators + 2 data instances, MAIN dynamic — resolve_main() picks it at call time; never hardcode an address).

Live Neo4j canonical migration (R4) — EXECUTED

scripts/bootstrap_graphs.py --neo4j --check now exits 0 on the live KG. It previously reported ~5.6k nodes lacking their canonical key. Investigation: the overwhelming majority were test pollution — tests that wrote real :Session/:Fact/:Memory/:AuditEntry nodes into the production graph over months (_TEST_…, test-sync-…, vizint-test-…, mod__test_…, /tmp/pytest-…).

scripts/migrate_neo4j_canonical.py (idempotent; run --dry-run first; take a backup — it deletes):

purges provably test-marked nodes (markers scanned against IDENTIFIER fields + the audit data blob only — NEVER free-text prose; scanning prose once deleted a real :Rule whose description said "start_test_services");
backfills the real/ambiguous remainder to unique canonical keys;
resolves duplicate-key groups without data loss (purge provable-test members; re-key survivors to legacy-<elementId>);
drops colliding plain indexes + wrong-key constraints; applies the canonical DDL.

Result of the live run: purged 5,124, backfilled/re-keyed 683; 766,445 → 761,321 nodes; real data intact; --check = 0. Idempotent (re-run = no-op).

Lesson: kg_schema.py check only flags MISSING keys, never DUPLICATE key values — apply() fails on duplicates separately, so the migration handles both. And classify test-vs-real on IDENTIFIERS, not prose.

Migration pipeline:

Rebuild rehearsal (R5.3) — `infra/rehearsal/`

A self-contained overlay rebuilds the full 13-node fleet from nothing on an isolated bridge (rhbr0, 10.93.0.0/24 — no collision with the live 10.53.1.x):

cd infra/rehearsal
./run_rehearsal.sh apply              # init+apply → provision → wait-ready → enroll → gen_env → bootstrap
./run_rehearsal.sh apply --with-suite # ... then the full suite
./run_rehearsal.sh teardown           # destroy all rh-* + rhbr0

terraform apply provisions all 14 resources from empty. Pieces: the golden image orchgraph-memgraph (the Memgraph fork binary), provision_node.sh (per-role static-IP + service install/start over the incus socket), enroll_orchgraph.py (Raft enrollment: ADD COORDINATOR ×2, REGISTER INSTANCE ×2, SET INSTANCE TO MAIN), and gen_env.py --tf-dir (derives a rehearsal .env from the overlay).

GREEN run (2026-07-11), measured — `rehearsal-timings.log`

Stage	Result
terraform apply	14 resources (13 containers + `rhbr0`) from nothing, all RUNNING
provision-fleet	all 13 nodes: static IP + service install/start, rc=0
wait-ready	10/10 bolt endpoints OPEN (redis + 4 neo4j + 3 coord + 2 data)
enroll-orchgraph	Raft HA green — leader + 2 followers, `instance_1` MAIN + `instance_2` replica, all `health=up`
bootstrap-neo4j	344s — canonical schema + full AST (16,710 nodes / 14,369 rels / 55 constraints) + architecture + spec/trace enrichment
memgraph round-trip	write+read via coordinator routing → MAIN (the paid full re-embed is byte-identical to the live-validated one; skipped to avoid duplicate OpenRouter cost)
infra-subset suite	142 passed / 0 failed in 193s against `10.93.0.x` — Neo4j sync+async, Memgraph adapter, Redis stores, Redis→Neo4j sync, Prolog governance

Rehearsal stage flow:

Why exec, not cloud-init: the live fleet was HAND-BUILT — even redis-queue reports NO-CLOUDINIT, and the golden image carries no cloud-init — and DHCP is broken on rhbr0. So provision_node.sh mirrors the real build: static IP over the incus socket, then per-role install. The orchgraph unit is rewritten with User=memgraph + a +-prefixed ExecStartPre because memgraph refuses to run as root when the data dir is memgraph-owned. See infra/rehearsal/README.md.

Drift gate

scripts/health_check.py::check_infra_drift runs terraform plan -detailed-exitcode in infra/terraform (exit 2 = the live fleet drifted from the IaC = fail). Skips cleanly where terraform/state is absent.

Terraform zero-diff recipe (lxc/incus, hard-won)

Per incus_instance: lifecycle { ignore_changes = [image, running] } (imported instances carry no recoverable image, which is ForceNew); a "name" key inside each nic properties; and NO explicit root disk device (Incus doesn't track the implicit root). terraform.tfstate is gitignored (local state).