Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0077

ADR-0077: Terraform owns every Incus environment; the live dev, test and prod fleets are imported, not rebuilt

← ADR-0076 · ADR-0078 →

The record

Status: Accepted (ruling, Mohamed, 2026-10-08: "all environments must be imported") · Date: 2026-10-08

Context

The Incus topology on the production host is described by three IaC layers that disagree, and by a shell script that is the one thing that actually built it. Measured on 2026-10-08 against origin/master b125d048f and the live daemon:

• infra/terraform declares the production fleet only (redis-queue, prolog-node-{1,2,3}, orchgraph-coord-{1,2,3}, orchgraph-data-{1,2}, prometheus-metrics). infra/README.md says the fleet was imported with a zero-diff plan on 2026-09-10 into a gitignored local terraform.tfstate. No state file exists anywhere on the host today (find / -name '*.tfstate*' over every filesystem: nothing). The definition is stale: its nic device names incusbr0, the live prod bridge is orch-prod-br (10.53.1.0/24). Last touched 2026-08-16.
• deploy/provision_environments.sh (last touched 2026-09-24) created every live project, bridge, ACL and container for dev, test and prod. It names INFRASTRUCTURE.md and env/{dev,test,prod}.env as its topology authority — not terraform.
• infra/rehearsal is a terraform overlay for the R5.3 disaster drill on an isolated bridge rhbr0 (10.93.0.0/24); infra/rawmachine is cloud-init for a fresh host. Neither describes the live fleets.
• The drift gate is silently off. scripts/health_check.py check_infra_drift returns PASS with a note whenever terraform.tfstate is missing, calls the terraform binary (1.5.7 on this host, which cannot even init a root whose versions.tf requires ≥ 1.6; tofu 1.9.0 can), and no timer runs the health check.
• Nothing owns four stale containers: rh-coord-{1,2,3}/rh-data-1 in project default on rhbr0 are the HA sandbox's pre-R4 placement (ruling R4, 2026-09-23, moved it to orchestrata-test 10.53.2.30–.33); the rehearsal overlay uses different names. Three prod nodes (prolog-node-{1,2,3}) and prometheus-metrics have been STOPPED since 2026-09-24; no container in any project sets boot.autostart.

The live inventory terraform must own (projects, bridges, ACLs, the two project-scoped default profiles that give test/dev containers their nic, and 23 containers) was dumped from the daemon on 2026-10-08 and every container's config is EMPTY apart from image.*/volatile.* — the same minimal shape infra/README.md already documents for zero-diff import.

Decision (Mohamed, verbatim: "all environments must be imported")

1. Terraform is the single source of truth for the Incus topology of all three environments — projects orchestrata-test and orchestrata-dev, bridges orch-prod-br/orch-test-br/orch-dev-br with their ACL bindings, ACLs orch-no-prod/orch-no-dev/orch-no-test, the two project default profiles, and every container in default (prod), orchestrata-test and orchestrata-dev. The shell provisioners keep only what terraform cannot express: service install and configuration inside a container, and Raft enrollment.
2. Everything live is IMPORTED, never recreated. Acceptance for the import is tofu plan -detailed-exitcode → 0 (no changes) on the live host after import.sh runs. The HCL models the as-built minimal shape (lifecycle { ignore_changes = [image, running] }, no explicit root disk, nic devices exactly as the daemon reports them).
3. State lives outside every checkout. The root's backend is local with path = "/var/lib/orchestrata/terraform/fleet.tfstate" — the host-owned state directory the install already bootstraps — so a worktree gc, a re-clone or a .gitignore can never lose it again (the September state was lost exactly that way). State backups land beside the existing /var/lib/orchestrata/backups.
4. The drift gate fails loudly. check_infra_drift runs tofu (falling back to terraform only when it satisfies required_version), reads the state path above, and a MISSING state is a FAIL, never a skip. The health check runs on a user timer so drift is detected, not discovered.
5. Out of scope, by name. rhbr0 and the four stale rh-* containers in default are retired, not imported (deletion is Mohamed's command). The orchestrata Incus project (codetools' SPEC-013 fleet) belongs to the codetools repo and is a separate question. incusbr0 and its unrelated containers are not Orchestrata's. infra/rehearsal keeps its isolated subnet by design; infra/rawmachine stays the fresh-host path.
6. First managed change after zero-diff: boot.autostart = "true" is declared for every imported container and applied, and the four stopped prod nodes are started — the import's acceptance is measured BEFORE this change so the two are never confused.

Alternatives rejected

• The shell provisioner as the declared authority, terraform retired to a drift gate: drift stays detectable only by the gate, and the gate just proved it goes quiet on a missing state file; the ruling asks for the fleets to be imported, not watched.
• Repair only (re-import prod, keep three layers): restores the 2026-09-10 state without deciding who owns the topology — the condition under which the state was lost once already.
• Terraform recreating the fleets from HCL instead of importing: destroys running production data nodes to prove reproducibility; the rehearsal overlay exists for that proof on an isolated subnet.
• State committed to the repo: a worktree branches off origin/master and would carry a stale copy; two checkouts applying against two states is the failure the local host path prevents.

Consequences

• infra/terraform gains projects.tf, networks.tf (bridges + ACLs), profiles.tf, test_fleet.tf, dev_fleet.tf; the prod resources are corrected to orch-prod-br; import.sh covers every resource; the backend block pins the host state path. Executed as task FLEET-IMPORT.
• provision_environments.sh and provision_test_fleet.sh must stop creating projects, bridges, ACLs and containers (a follow-up task): they refuse when terraform owns the name, and converge only the in-container layer.
• health_check.py check_infra_drift is rewritten per item 4 with tests; a deploy/systemd timer unit runs it.
• The rehearsal overlay (infra/rehearsal) is unaffected; its enroll_orchgraph.py stays the shared enrollment script.
• A security finding surfaced by the inventory, recorded here for a separate decision: profile mcp-server-test-ssh in project default embeds an SSH private key in its cloud-init user.user-data.

Provenance

Mohamed, 2026-10-08, in the Claude Code session (Fable 5.1), after the orchestrator researched why the four stale rh-* containers were still on his plate and presented three positions (terraform authoritative for all fleets — recommended; shell provisioner authoritative; repair only). His ruling, verbatim: "all environments must be imported". Evidence: live daemon dump 2026-10-08 (incus list --project {default,orchestrata-test, orchestrata-dev} -f json, networks, projects, ACLs, profiles); find for state files over all filesystems (none); scripts/health_check.py:493-520 (skip-on-missing-state); infra/terraform/coordinators.tf (incusbr0); deploy/provision_environments.sh header (topology authority); ruling R4 in deploy/orchestrata-test/provision_test_fleet.sh:391-399.

Related ADRs

No other ADR is named in this record.

Rulings cited

None.

Specs cited

• SPEC-013