Orchestrata Ultimate › Plans › reproducibility-plan

Reproducibility & Infrastructure-as-Code Plan

Source: plans/reproducibility-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-10 · 257 lines.

Prose pending.

Plan text

Reproducibility & Infrastructure-as-Code Plan

Drafted 2026-07-10 (session 224) from a live fleet audit (incus list, per-container incus config show, in-container version probes). Goal: a fresh machine — or this machine after a disk loss — reaches "full suite green" from git clone + one bootstrap command, with zero undocumented hand-built state.

1. Ground truth — as-built (audited 2026-07-10; re-audited + R1 applied 2026-07-11)

18 RUNNING Incus (LXD 7.2) containers on bridge network 10.92.53.0/24. The suite depends on the starred rows. The memgraph-1/2/3/-ha/-db names in the 2026-07-10 draft were STALE — that named cluster no longer exists; the live Memgraph HA cluster is the orchgraph-* fleet below (matches docs/LEVEL0.md). All starred rows are now PINNED (R1, 2026-07-11).

The two findings that motivated this plan

1. DHCP-lease IP fragility (RESOLVED by R1, 2026-07-11). The two most load-bearing addresses in .env — Redis .10 and Neo4j .11 — and the ENTIRE orchgraph/Memgraph HA fleet held their IPs only by lease longevity. R1 pinned all 8 to their current IPs via incus config device override <c> eth0 ipv4.address=<ip> and restart-tested each one at a time: every IP persisted.
2. Memgraph HA MAIN is dynamic — confirmed live. Restart-testing the MAIN data instance during R1 triggered a real failover (MAIN moved instance_2 .123 → instance_1 .128; the cluster recovered to 5/5 after the old MAIN finished its ~90s vector-index WAL recovery). Consumers MUST discover MAIN via the HA controller / SHOW INSTANCES on a coordinator, never by pinned address — the whole point of R3. (Also observed: many leftover test_chunk_idx__TEST_* vector indexes slow recovery — test hygiene debt, tracked separately.)

2. Phase R1 — pin what must not move (½ day, do first)

For every suite-critical container, make the leased IP the configured IP (idempotent; no renumbering, so nothing else changes):

for c in redis-queue:10.92.53.10 neo4j-db:10.92.53.11 \
         memgraph-1:10.92.53.111 memgraph-2:10.92.53.192 \
         memgraph-3:10.92.53.159 memgraph-ha:10.92.53.153 \
         memgraph-db:10.92.53.174 claude-orchestrata:10.92.53.12; do
  name=${c%%:*}; ip=${c##*:}
  incus config device override "$name" eth0 ipv4.address="$ip"
done

Acceptance: incus config show <c> --expanded | grep ipv4.address shows the pin for all 8; containers restart-tested one at a time; suite stays green.

3. Phase R2 — .env.example + config contract (½ day)

• Commit .env.example with every key the code reads (audited from .env 2026-07-10): REDIS_HOST/PORT/DB, NEO4J_URI/USERNAME/PASSWORD, NEO4J_URI_GW1..GW7, MEMGRAPH_URI/URIS/USERNAME/PASSWORD, MEMGRAPH_HA_CONTROLLER_URL, MEMGRAPH_CHUNK_LABEL, MEMGRAPH_VECTOR_INDEX_NAME, CXP_GRPC_HOST/PORT, KG_GRPC_PORT, DEEPSEEK_API_KEY, OPENROUTER_API_KEY, GITLAB_TOKEN — placeholder values, one comment per key naming the consumer module.
• Rule stays: values read at call time via os.environ.get() (no import-time constants, no hardcoded port defaults — raise if unset).
• Add a scripts/validate_env.py preflight (or extend start_test_services.py): fail fast, listing missing keys, before any test infra work. CI and local run the same check.

4. Phase R3 — Memgraph MAIN discovery (1 day)

• semantic_graph/ha_controller.py already speaks to the controller; route ALL writers (wiki-rag-ingest, scripts/ingest_orchestrata_ultimate.py, scripts/import_neo4j_to_memgraph.py) through a resolve_main() helper: ask MEMGRAPH_HA_CONTROLLER_URL, fall back to probing each of MEMGRAPH_URIS with SHOW REPLICATION ROLE, cache for the process.
• Delete every doc/wiki assertion that names a specific node as MAIN; replace with "discover via controller".
• Test: existing tests/semantic_graph/test_memgraph_ha.py + test_ha_replica_health.py extended with a resolve_main() contract test against the live cluster.

5. Phase R4 — graph state is DERIVED, never hand-built (1–2 days)

Both graphs must be rebuildable from source + seeds:

• Neo4j (operational KG) — one idempotent bootstrap entry point (scripts/kg_schema.py apply already applies SPEC-106 DDL):
  1. constraints/indexes: kg_schema.py apply (idempotent MERGE/IF NOT EXISTS);
  2. code intelligence: scripts/sync_ast_to_neo4j.py;
  3. architecture rules: scripts/sync_architecture_to_neo4j.py;
  4. specs/RTM: scripts/enrich_semantic_graph_specs.py (spec side). Wrap 1–4 as scripts/bootstrap_graphs.py --neo4j with a --check mode that diffs live schema vs canonical (canonical_schema.py).
• Memgraph (semantic/RAG) — rebuild = scripts/ingest_orchestrata_ultimate.py
  • wiki-side wiki-rag-ingest (owned by knowledge-wiki). Respect the shared-cluster discipline: label-scoped, never touch :Entity nodes this project didn't create; ontology (SPEC-106 canonical_schema + kg-schema-gaps) is designed BEFORE any import.
• Per-worker Neo4j replicas (.13/.14/.19): document their creation as incus copy neo4j-db neo4j-node-N + password reset + NEO4J_URI_GW<N> registration — scripted in the IaC layer below.

6. Phase R5 — Infrastructure as Code (the detailed plan)

Tool choice: Terraform + the official lxd/incus provider for container lifecycle, plus cloud-init per container for in-guest provisioning. Rationale: declarative state diffing (terraform plan shows drift — the exact failure mode we just audited by hand), the provider supports ipv4.address pins, profiles, and limits natively; no agent runtime needed on guests. Ansible is the fallback if in-guest config grows beyond cloud-init (Neo4j clustering, Memgraph HA enrollment already flirt with that line — start with cloud-init, promote to Ansible roles only when a guest needs >1 config file templated).

Layout (new top-level infra/):

infra/
  terraform/
    versions.tf          # incus provider, pinned version
    network.tf           # incusbr0 10.92.53.0/24, DHCP range excluding pins
    profiles.tf          # base-debian (bookworm), base-ubuntu (noble), limits
    redis.tf             # redis-queue: image, pin .10, cloud-init installs redis 7.x
    neo4j.tf             # neo4j-db .11 + count=3 worker replicas .13/.14/.19
    memgraph.tf          # 3 HA nodes + controller + (legacy) single node
    prolog.tf            # 3 nodes, 4CPU/4GB limits, swipl 9.x
    outputs.tf           # every service IP:port → consumed by .env generation
  cloud-init/
    redis.yaml neo4j.yaml memgraph.yaml prolog.yaml
  gen_env.py             # terraform output -json → .env (bridges IaC → app config)
  README.md              # bootstrap: terraform apply && python infra/gen_env.py
                         #            && python scripts/bootstrap_graphs.py --all

Sequenced adoption (never big-bang):

1. R5.1 import, don't recreate — terraform import each running suite-critical container into state; terraform plan must converge to zero-diff before any apply is trusted. (~1 day)
2. R5.2 drift gate — CI job (and health_check.py step) runs terraform plan -detailed-exitcode; non-zero = infra drifted from code = red. (~½ day)
3. R5.3 rebuild rehearsal — on a scratch Incus project (incus project create rehearsal), terraform apply from nothing + gen_env.py + bootstrap_graphs.py + full suite. The rehearsal IS the acceptance test; document wall-clock. (~1 day, repeatable)
4. R5.4 secrets — API keys/passwords never in tfvars committed to git: secrets.auto.tfvars gitignored; gen_env.py merges them from ~/.config/orchestrata/secrets.env (0600). GITLAB_TOKEN stays OUT of generated .env (the expired-token-shadows-glab incident).
5. R5.5 versions — service versions pinned in cloud-init (redis=7.0.x, neo4j=5.26.x, memgraph latest-noble, swipl=9.0.x); upgrades are PRs that change a pin, run the rehearsal, then apply to prod containers one node at a time.

7. Phase R6 — CI / local runner parity (1 day)

• One entry point everywhere: python scripts/run_tests.py (infra check → blocking gates → batched executor → solo-retry round). CI must not invent its own pytest invocation. scripts/validate_ci.py already gates config drift — extend it to assert the CI job calls the canonical runner.
• Runner needs only .env + reachable fleet: a self-hosted runner on this host (or any host after R5.3 rehearsal) is sufficient; no GitLab-managed runner can reach 10.92.53.x.
• Cache uv + .venv keyed on uv.lock; Python pinned 3.14t via .python-version (already committed).
• Publish .flake_recoveries.ndjson as a CI artifact — recoveries are the flake-hunting signal, and CI is where recurrence across runs becomes visible.

7b. As-executed status (session 225, 2026-07-11)

• R1 — DONE. All 8 unpinned suite-critical containers pinned to their live IPs and restart-tested (IPs persisted; a MAIN restart proved HA failover).
• R2 — DONE + pushed. .env.example (98 keys) + scripts/validate_env.py preflight, wired into start_test_services.py.
• R3 — DONE + pushed. semantic_graph/main_discovery.resolve_main; all writers route through the adapter chokepoint; stale MAIN doc-pins reframed.
• R4 — DONE + pushed; live Neo4j migration EXECUTED (session 226, 2026-07-11). scripts/bootstrap_graphs.py (--neo4j/--memgraph/--all/--check/--plan); --check exits 0 on Memgraph. The LIVE Neo4j legacy drift is now CLEARED: scripts/migrate_neo4j_canonical.py (backed by a full rollback backup) purged 5 124 test-pollution nodes (tests that had written _TEST_/test-sync-/vizint-test-/mod__test_ nodes into prod over months), backfilled/re-keyed 683 genuinely-real or ambiguous nodes to unique canonical keys, dropped the 2 colliding plain indexes + 5 wrong-key constraints, and applied the 26 canonical constraints. --check now exits 0 on the live operational KG (766 445 → 761 321 nodes; real data — arch Rules, Architecture, Volks, CodeFunctions, 256k sessions — intact). The migration is idempotent (re-run = no-op) and tested (tests/scripts/ test_migrate_neo4j_canonical.py, 16/16). Strategy (purge-provable-pollution + backfill-the-rest) was operator-approved.
• R5.1/R5.2 — DONE + pushed. All 13 suite-critical containers imported into Terraform; terraform plan is zero-diff. Drift gate (terraform plan -detailed-exitcode) wired into scripts/health_check.py (check_infra_drift).
• R5.3 rehearsal — GREEN end-to-end (session 227, 2026-07-11). infra/rehearsal/ — a self-contained overlay — rebuilds the FULL 13-node fleet from nothing on an ISOLATED bridge (rhbr0, 10.93.0.0/24, no collision with live), then runs the app suite green against it, all with measured wall-clock (rehearsal-timings.log). Pieces: the golden orchgraph image (orchgraph-memgraph, /opt/memgraph fork binary), provision_node.sh (per-role static-IP + service install/start over the incus socket), enroll_orchgraph.py (Raft enrollment from the captured live topology), the run_rehearsal.sh harness, and gen_env.py --tf-dir. Green run: terraform apply (14 resources from empty) → provision 13 nodes → 10/10 bolt endpoints → Raft HA (MAIN + replica + 3 coords, all health=up) → gen_env → bootstrap-neo4j (16,710 nodes, 344s) + memgraph coordinator-routed round-trip → infra-subset suite 142 passed / 0 failed in 193s against 10.93.0.x. Why exec, not cloud-init: the live fleet was HAND-built — even redis-queue reports NO-CLOUDINIT, the golden image carries no cloud-init — and DHCP is broken on rhbr0; provision_node.sh mirrors the real build (static IP over the incus socket, then per-role install; the orchgraph unit is rewritten with User=memgraph + +-prefixed ExecStartPre because memgraph refuses to run as root when its data dir is memgraph-owned). Full detail: infra/rehearsal/README.md.
• R6 — DONE + pushed. CI repointed to the canonical run_tests.py; validate_ci gate enforces it; .python-version committed.

8. Acceptance criteria (the plan is DONE when)

1. All 8 suite-critical unpinned containers carry ipv4.address pins and survive a restart with the same IP (R1).
2. .env.example committed; validate_env preflight red on any missing key (R2).
3. No consumer writes to Memgraph by pinned MAIN address; resolve_main() contract test green against the live cluster (R3).
4. bootstrap_graphs.py --all --check exits 0 on a freshly rebuilt pair of graphs; both graphs demonstrably rebuild from source (R4).
5. terraform plan zero-diff on the imported fleet; drift gate wired into CI + health_check (R5.1–R5.2).
6. One documented rehearsal from empty Incus project to full-suite green, with measured wall-clock (R5.3).
7. CI and local both run scripts/run_tests.py; validate_ci enforces it (R6).

9. Sequencing & effort

Total ≈ 7–9 focused days. R1+R2 are the cheap insurance and should land this week regardless of when R5 starts.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks