Home / Infrastructure / Incus fleets

Incus fleets

Every container orchestrata uses runs in Incus on this host, openmandriva-x8664. They are grouped into five Incus projects. Three of them are the environments of ADR-0077: prod, test and dev. Each environment has its own bridge and subnet, and network ACLs keep the environments from reaching each other. The fourth project, orchestrata, holds the wiki, a Claude workspace container, a stopped GitLab server and codetools test leftovers. The fifth, pi-workers, exists for pi-task and holds no instances.

All state on this page was measured on 2026-10-23 between 01:15 and 01:25 EDT with read-only commands:

• incus list --all-projects (state, IPv4, image.os, boot.autostart);
• incus project list, incus network list, incus storage list;
• incus config show knowledge-wiki;
• incus exec app -- cat /opt/orchestrata/VERSION;
• /var/lib/orchestrata/terraform/drift-latest.json;
• infra/README.md and infra/terraform/*.tf in wt-MAP-PRECOMMIT at 8a2cb42ad.

Topology

The environments

Autostart on the test and dev fleets was turned off by hand, to cut CPU load. Mohamed confirmed on 2026-10-22 that he did it; the record of that day dates the change 2026-10-20. On 2026-10-21 at 19:32, a DeepChat session stopped the test and dev fleets. At 01:20 on 2026-10-23, every container in the three environments was running again.

Nothing starts at boot, prod included (ADR-0135, Mohamed 2026-10-25: "deliberate, keep off"). All 30 containers in default, orchestrata, orchestrata-dev and orchestrata-test carry boot.autostart=false. The orchestrata user units are disabled too: a systemctl run from a Konsole window removed their default.target.wants links at 2026-10-24 11:34:03. So after a reboot nothing runs until someone starts it. On 2026-10-24 the T9 drive that holds the pool fell off USB at 18:33:40, and the host rebooted four times. Prod stayed down until it was restored by hand on 2026-10-25: containers started at 10:38, units active at 10:42:23.

orch-up is the one command that starts the system (ADR-0135; built under brief plans/briefs/system-up.md):

• It refuses unless the T9 filesystems are mounted.
• It starts the prod containers, waits for Redis PING and a routed graph read, then starts the five prod units and checks each unit's interpreter and working directory.
• It then starts the test fleet and waits for start_test_services.py --status to report all_required_ok.
• It then starts the knowledge-wiki container.
• --with-dev adds the dev fleet; --check reports and changes nothing.

Prod: Incus project default

The three prolog-node-* containers that the OpenTofu files still declare were deleted on 2026-10-20 (see Drift). The cluster itself is described on Memgraph HA cluster reference.

Test: Incus project orchestrata-test

The product inside app is at 22363abb2, deployed 2026-10-12T13:20:13+00:00. That is 546 commits behind origin/master. See Deployment inventory.

Dev: Incus project orchestrata-dev

Incus project orchestrata

Seven of the ten containers in this project are stopped test leftovers. They are listed under codetools blockers.

Who owns what

• OpenTofu owns the topology of the three environments. infra/README.md (status 2026-10-08, ADR-0077) says: "ALL THREE environments imported, plan is ZERO-DIFF, drift gate live". The resources are 2 projects (orchestrata-test, orchestrata-dev; prod is the built-in default), 3 bridges, 3 ACLs (orch-no-prod, orch-no-test, orch-no-dev), 2 project default profiles and the instances. The instances are declared in coordinators.tf, data_nodes.tf, redis.tf, prometheus.tf, prolog.tf, memgraph_test.tf, test_fleet.tf and dev_fleet.tf.
• The state file is host-owned: /var/lib/orchestrata/terraform/fleet.tfstate. It is "never inside a checkout - the September state was lost to a worktree gc". It was last written on Oct 11.
• Shell provisioners do what OpenTofu cannot express. deploy/provision_environments.sh and provision_test_fleet.sh install and configure services inside containers and enrol Raft members. provision_app_container.sh copies the product into the test app.
• Nothing declares the orchestrata and pi-workers projects. No .tf file names knowledge-wiki, claude-orchestrata, gitlab-server or pi-workers.
• Storage. There is one storage pool, t9-btrfs, on the T9 disk. It must stay on simple quotas; see the T9 btrfs qgroup runbook.

Drift

The drift gate is scripts/check_infra_drift.py. It runs tofu plan -detailed-exitcode over infra/terraform and checks three bridge keys that the provider never stores. It writes /var/lib/orchestrata/terraform/drift-latest.json. A missing state is a failure, never a skip.

The last report was written on 2026-10-22 at 19:01:27 UTC:

• verdict: drift, plan_exit: 2, bridge_drift: [];
• the plan was "3 to add, 14 to change, 0 to destroy".

The three additions are incus_instance.prolog["node-1".."node-3"], which were deleted on the host. The in-place changes were measured on 2026-10-22 with a read-only tofu plan -refresh-only. There are 13 boot.autostart changes from "true" to "false" (3 dev, 9 test and test_app), plus the test project's restricted.devices.disk.paths. Mohamed made these changes by hand to cut CPU load. Reconciling infra/terraform with them is pending.

orchestrata-infra-drift.timer has been disabled since 2026-10-22, so this report is not refreshed. See section 10 of Host services.

Backups

orchgraph-backup.timer runs every day at 03:00. It takes a CREATE SNAPSHOT of every prod orchgraph database and pulls it to /var/lib/orchestrata/backups/orchgraph, keeping 14 runs. On 2026-10-23 the archive held 14 runs, from 20261009-030051 to 20261022-030027, 4.3 GB in all. The latest manifest records the node counts memgraph 19,973 and orchkg 156,953, with MAIN at 10.53.1.123:7687. Nothing backs up the test or dev fleets or the orchestrata project; none was found.

Incus snapshots and image copies follow Mohamed's one-generation rule: keep one generation only, and ask before touching images.

Rules

• IPs live only in the env profiles. Source and tests read REDIS_HOST, KG_URI and the rest from the environment at call time, never from a hard-coded address.
• Prod is never a test target. The test suite's authoritative target is orchestrata-test.
• The ACLs are the isolation. orch-no-prod, orch-no-test and orch-no-dev stop one environment's containers from reaching another's bridge.

Related

• Incus containers
• Memgraph HA cluster reference
• Redis, Gateway, Kernel, Wiki.js
• Deployment
• Host services
• Deployment inventory
• T9 btrfs qgroup runbook