Orchestrata Ultimate › Plans › recorded-run-2026-10-08-eight-failures-design

Recorded run 2026-10-08 — the eight failures: investigation, design, swarm plan

Source: plans/recorded-run-2026-10-08-eight-failures-design.md at 8a2cb42ad · last changed 2026-10-08 · first committed 2026-10-08 · 299 lines.

Prose pending.

Plan text

Recorded run 2026-10-08 — the eight failures: investigation, design, swarm plan

Run: promote-test-irc, HEAD 5cc8bbd5e, 1832 passed / 3 failed / 5 batch-failed-solo-recovered, 1429.7s. Tasks filed: 572–579 (session 271). Hot-fixes shipped: 3bb5319df, 0fa0ac048 (master 1d5f3e52c, prod promoted, recorded run 38599 passed / 0 failed). Why this document exists: the hot-fixes were dispatched under DESIGN_WAIVER (RUL-022). This is the design they lacked, written from the AST toolkit and the live host, plus the structural fixes for what the symptoms were pointing at — tasks 580 and 581 are open, and three of the eight were symptoms of config, deploy and test-generation defects that a per-test patch cannot close. Contract diagram (the RUL-022 artifact every builder brief names): plans/contracts/eight-failures-2026-10-08.mmd. Convention: worktree-swarm per plans/arch-eval-remediation-swarm-plan.md (increment & verification protocol, hard rules) — inherited verbatim below.

1. The eight failures at a glance

Three of the eight asserted on host state (an env key, an installed unit, a directory in cwd) rather than behaviour; two were a load race in a fixed budget; one was a genuine architecture regression whose hot-fix moved the defect into the deploy; one was a performance defect half-fixed. The work items below close the causes.

2. Investigation record (what was queried, what it said)

All structure/content below came from the AST toolkit on the shipped tree (wt-IRC-FLEET @ 1d5f3e52c, snapshot refreshed in c4597f459), tofu output -json over the host state, and git log. Nothing is argued from memory.

2.1 Config / IaC (failure 1)

• infra/terraform/versions.tf: ADR-0077 (2026-10-08) — terraform owns all three environments; state at /var/lib/orchestrata/terraform/fleet.tfstate; drift gate scripts/check_infra_drift.py runs hourly (deploy/systemd/orchestrata-infra-drift.timer). drift-latest.json: verdict: zero-diff at 12:01 UTC today.
• tofu output -json test_fleet_ipv4 → {"app":"10.53.2.13","memgraph":"10.53.2.10","redis":"10.53.2.11","prolog":"10.53.2.12","memgraph-ha-main":"10.53.2.20","memgraph-ha-replica":"10.53.2.21","rh-coord-1..3":"10.53.2.30-32","rh-data-1":"10.53.2.33"} — these match env/test.env today, but test_fleet.tf:6 says: "Live addresses (DHCP leases on orch-test-br, NOT pinned on the devices) … Only app pins its address … Pinning the rest is a managed change after zero-diff."
• infra/gen_env.py:61-84 env_lines(outputs) emits only REDIS_HOST/PORT, PROLOG_NODES, MEMGRAPH_URI(S) from the prod outputs; there is no profile argument and no TEST_* derivation. outputs.tf:38-40 says test_fleet_ipv4 is "the truth gen_env.py / env/{test,dev}.env must agree with" — nothing enforces it.
• .env.example is tracked (git ls-files), hand-edited (its TEST_MEMGRAPH_URI drifted to a prod-subnet address for ~2 months). scripts/check_env_subnet_scheme.py checks env/*.env subnets, not the template, and cannot see a commented-out key.
• infra/terraform/memgraph_test.tf + var.memgraph_test_ip are retired in prose ("Mohamed: git rm infra/terraform/memgraph_test.tf closes it") — the file still exists.

2.2 Tools deploy closure (failure 3 residue, task 580)

• deploy/install-tools.sh:120-135: git archive vendors src/orchestrata_ultimate/{__init__,core,knowledge_graph,refactor} + orchestrata_ast and nine scripts; the venv gets neo4j watchfiles fastapi uvicorn pydantic psutil xxhash.
• Computed with a static walker (scratch import_closure.py, same algorithm WI-DEPLOY productises):
  • eager closure of refactor.codemod._daemon_entry: 1 module (everything in the daemon is lazy) — this is why the deploy smoke passes.
  • closure of _daemon_entry + review_checks (what the first InMemoryReviewEngine(registry=None) executes): 103 modules — core 40, refactor 34, verification 14, openspec 7, knowledge_graph 2, orchestrata_ast 5.
  • all-scopes closure of every deployed entry (+ pytest_live_reader/observer, knowledge_graph.sync): 168 modules, adding providers 16, llm 5, memory 7 (lazy paths).
  • orchestrata_ultimate.refactor.codemod.persistence has no __init__.py (namespace package by accident).
• who-imports review_checks → only review_engine.py:190 (the deferred import). who-imports verification.review.checks_safety → review_checks.py:49, orchestrator/verification_gate.py:70, verification/review/engine.py:18.

2.3 Feature-loss gate (failure 7, tasks 578/581)

• scripts/check_feature_loss.py:1437-1541 run_check: collect_current_implementation (Python harvest_src_declarations + TS/JS harvest) → harvest_test_citations(tests_dir) → find_deleted_symbols → … → find_surface_losses.
• scripts/generate_rtm_evidence.py:338-405 harvest_test_citations: per test module read_text → REQ pre-filter (shipped) → _claim_positions (ast.parse + tokenize) → then collect_traces(str(tests_dir)) (core/tracing/scanner.py:184-211) walks and ast.parses every module again, with no pre-filter. Both forms of citation contain the literal REQ-, so one pre-filter is valid for both.
• Measured 2026-10-08 (same loaded box): harvest 165.1s → 63.0s after the pre-filter; remaining budget consumers: second parse walk, collect_current_implementation ≈ 10s, find_surface_losses ≈ 9s.

2.4 Generated tests write into the checkout (failure 4 residue, task 575)

• test_value/test_value.jsonl at five checkout roots; content: {"event": "trace_start", "iteration_id": "test_value", …} → core/tracing/__init__.py:53 IterationTracer writes os.path.join(output_dir, f"{iteration_id}.jsonl").
• Generators that emit "test_value" for every str parameter: orchestrata_tools/test_generator/{enhanced_generator.py:40, jinja_gen.py:184, stub_generator.py:22, testgen/types.py:35}. The generated autofix tests call autofix_relative_imports(workspace="test_value") — a non-existent workspace — so they verified nothing until the stray dir made them pass.
• tests/plugins/git_sentinel.py (live-repo mutation sentinel) did not flag it: an untracked new directory is invisible to a diff-based sentinel.

2.5 Promotion unit filter (failure 2)

• promotion/stages_units.py:68-126: four filters mirror install.sh; the fourth runs systemctl --<scope> cat <dep> for every Requires=/BindsTo= name. The shipped test proves the filter with a synthetic name, but the probe remains host-coupled and unseamed.

2.6 Executor budgets (failures 5/6)

• tests/core/threading/_batch_budget.py:65-81 calibrated_batch_budget runs ParallelTestExecutor(max_workers=1) on the tiny fixture, budget = clamp(4 × boot, 5.0, 30.0), child env ORCHESTRATA_SLOW_FIXTURE_SLEEP_S = budget + 4. The production runner derives per-batch budgets from duration history (test_scheduler.py), so no src change is warranted.

3. Design

Composition over inheritance; Protocols are the only interfaces; every new class composes an injected Tracer. The class contract is the .mmd above; this section is the rationale and the non-class parts (HCL, shell, gates).

3.1 WI-ENV — config derived from IaC, agreement enforced

Problem. Three artifacts claim the test-fleet endpoints (env/test.env, .env.example, the terraform outputs); none derives from another; the only thing that noticed two months of drift was a test that happened to need the key.

Design.

1. Pin the test-fleet addresses in terraform (the "managed change after zero-diff" test_fleet.tf anticipates): a var.test_fleet_ips map {memgraph="10.53.2.10", redis="10.53.2.11", prolog="10.53.2.12", memgraph-ha-main="10.53.2.20", memgraph-ha-replica="10.53.2.21", rh-coord-1..3, rh-data-1} and a per-instance device { name="eth0" type="nic" properties={ "name"="eth0", "network"=incus_network.test.name, "ipv4.address"=each.value } } on incus_instance.test. The values are the live leases, so the plan is a device add with no address change; the lease becomes a static assignment.
   • Research (incus provider, 2026-10): a nic ipv4.address set through the provider is the same operation as incus config device set <inst> eth0 ipv4.address=…, which Incus applies on the next restart; whether the provider plans it as in-place or replacement must be read from tofu plan — a replacement plan is a ⛔ stop (never destroy a running data node to pin an address; fall back to incus config device set + lifecycle.ignore_changes on that device). Sources: static IPv4 for instance with the terraform incus provider, ipv6.address none on an instance nic.
   • Not applied by the swarm. The worker produces the HCL and the tofu plan -detailed-exitcode output (expected exit 2, in-place device adds only). Apply is a central step after Mohamed reads the plan (§6).
2. Outputs (outputs.tf): test_fleet_endpoints = { MEMGRAPH_URI="bolt://<memgraph>:7687", MEMGRAPH_URIS=…, TEST_MEMGRAPH_URI=…, REDIS_HOST, REDIS_PORT="6379", PROLOG_NODES, CXP_GRPC_ENDPOINT="<app>:50053", GATEWAY_HOST=<app>, TEST_MEMGRAPH_HA_MAIN_URI, TEST_MEMGRAPH_HA_REPLICA_URI } and the same shape for dev_fleet_endpoints; prod keeps the existing outputs.
3. infra/gen_env.py becomes profile-aware: TerraformOutputs (dataclass over tofu output -json, built through an injected CommandRunner so tests feed it the JSON text, never a subprocess double), FleetProfile enum, EnvDeriver.env_lines(profile) and .template_lines(profile); CLI gen_env.py --profile test --write env/test.env, --template --write .env.example (updates only the keys it owns, via the existing write_into_env stale-line replacement). Existing prod behaviour unchanged (default profile prod).
4. Gate scripts/check_env_fleet_agreement.py (FleetAgreementGate): for each profile, derive the lines and compare against the committed env/<profile>.env and the template's TEST_* keys; any mismatch, any commented-out owned key, missing tofu, missing state → FAIL (ADR-0077 item 4 discipline). Wired into scripts/health_check.py beside check_infra_drift and into deploy/ship.sh's verify step.
5. Retire infra/terraform/memgraph_test.tf and var.memgraph_test_ip: git rm is Mohamed's command per the file header — listed in §6, not done by the swarm.
6. Tests (real tofu output -json fixtures captured from the host, no doubles): tests/infra/test_gen_env_profiles.py, tests/scripts/test_check_env_fleet_agreement.py; tests/core/test_test_memgraph_routing.py::test_env_template_names_the_test_fleet_node_for_the_suite stays and now also asserts the template equals the derivation.

3.2 WI-DEPLOY — the tools deploy vendors a computed closure and proves a review

Problem. install-tools.sh lists four subtrees by hand. The daemon imports in 1 eager module; its first real review needs 103 modules across six packages. The smoke tests "imports", not "works".

Design.

1. scripts/tools_import_closure.py (ImportClosure + EagerImportCollector / AllScopesImportCollector strategies → ClosureReport): --entries (default: the deployed entry modules), --eager, --json; reports modules, top-level packages, third-party imports (so the venv uv pip install line is computed too), unresolved modules.
2. deploy/tools-vendor-manifest.txt: the committed list of src/orchestrata_ultimate/<pkg> subtrees + orchestrata_ast; install-tools.sh reads it for git archive instead of the inline list. A test (tests/deploy/test_tools_vendor_manifest.py) asserts manifest == ClosureReport.vendor_manifest() of the all-scopes closure of the deployed entries, so a new lazy dependency fails in CI, not at deploy.
3. Deploy smoke (install-tools.sh verification): construct InMemoryReviewEngine() (registry=None) in the deployed venv and run review over a two-file throwaway workspace; the result must be a list. This is exactly the path that fails today.
4. refactor/codemod/persistence/__init__.py added (namespace-package accident; importlib resolution in the deployed lib must not depend on it).
5. Third-party deps the closure introduces (verification/openspec) go into the install line as measured by the script; the installer comment "measured against a clean venv, not assumed" is kept true.

3.3 WI-FLOSS — one parse per test module, shared by both harvesters

Design.

1. core/tracing/scanner.py gains ParsedModule and ModuleCorpus.load(root, filter=RequirementMentionFilter()) — one read_text + one ast.parse per module that passes the filter; collect_from(corpus) is the corpus-driven twin of collect_traces(root_dir) (which stays, implemented as collect_from(ModuleCorpus.load(root, filter=AcceptAll())) so behaviour is byte-identical for every existing caller).
2. generate_rtm_evidence.harvest_test_citations(tests_dir, corpus=None): when a corpus is given, the docstring/comment pass reads pm.text and _claim_positions takes the tree (tokenize still needed for comment columns — keep it, but only for modules in the corpus), and the decorator pass uses TraceScanner.collect_from(corpus). Without a corpus it builds one itself (so generate_rtm_evidence.py as a CLI is unchanged).
3. check_feature_loss.run_check builds the tests corpus once and passes it to both; collect_current_implementation and find_surface_losses are profiled in part 1 and, if they re-parse src/, get the same corpus treatment over src/.
4. Acceptance: output identical (365 ids, compare_harvest.py style diff), canary < 60s under a concurrent run_tests_streaming.py tests/scripts -j 8 load on this box, and the 120s pytest budget kept (not raised).

3.4 WI-TESTGEN — generated tests never write into the checkout

Design.

1. ParamValueStrategy Protocol in orchestrata_tools/test_generator/testgen/types.py; PathLikeParamStrategy for parameter names in {workspace, output_dir, out_dir, root, repo_root, path, file_path, directory, *_dir, *_path, *_root} emits str(tmp_path / "<name>") and the generated test gains a tmp_path fixture parameter; LiteralParamStrategy keeps today's literals for everything else. All four generators consume the strategy (one source of truth; the duplicate literal tables collapse into types.py).
2. Regenerate the affected generated modules with scripts/generate_pytest.py --package core / --package recovery scope (tests/generated_core/test_tracing___init___generated.py, tests/generated_recovery/test_verification_autofix_{import_fixes,engine}.py); diff must show only the path-literal change.
3. tests/plugins/git_sentinel.py gains UntrackedRootPathDetector: snapshot the repo root's untracked top-level entries before the session and after each module; a new entry fails that module with its name — the class of pollution that produced test_value/ becomes a red test, not a cwd accident.
4. The five stray test_value/ directories are not deleted by the swarm (user ruling: nothing is deleted); §6 lists them for Mohamed.

3.5 WI-UNITS — the promotion unit filter asks a probe

Design. UnitProbe Protocol (loads(unit, scope) -> bool), SystemdUnitProbe (today's systemctl --<scope> cat), DirectoryUnitProbe(unit_dirs) (loads iff a file <name> exists under one of the dirs — a real implementation, the offline answer install.sh --dry-run would want); daemon_units(target_root, probe: UnitProbe | None = None) defaults to the systemd probe so production is unchanged. Tests cover both probes; the shipped synthetic-name test stays as the host-level proof. Low priority — tranche 2.

3.6 Recorded designs — shipped, no further build

• Batch budget calibration (BatchBudgetCalibrator in the diagram = tests/core/threading/_batch_budget.py): measured boot × 4, floor 5s, ceiling 30s, slow fixture driven past the budget by env. Rationale: a SIGKILL scenario must be slower than boot by a margin that scales with the host, not a constant.
• Cycle leaf review_context.py + ReviewableWorkspace Protocol: the only way to remove a cycle the ratchet counts across all scopes is to move shared types to a leaf; the engine annotates with the Protocol, never imports workspace.
• Autofix guard _detect_flat_package: a missing workspace is "no package", not an exception.

4. Terraform / config design (the IaC half of WI-ENV)

Diagram source shown as text: Mermaid could not parse it (Error: Lexical error on line 4. Unrecognized text.).

flowchart LR
    TF[infra/terraform<br/>test_fleet.tf · variables.tf · outputs.tf] -- tofu output -json --> OUT[(TerraformOutputs)]
    OUT --> DER[EnvDeriver<br/>profile=test/dev/prod]
    DER -- --write --> ENV[env/test.env · env/dev.env]
    DER -- --template --> TPL[.env.example TEST_* keys]
    ENV & TPL & OUT --> GATE[check_env_fleet_agreement.py<br/>FAIL on mismatch / commented key / no state]
    GATE --> HC[health_check.py · ship.sh verify]
    STATE[(/var/lib/orchestrata/terraform/fleet.tfstate)] --> OUT
    DRIFT[check_infra_drift.py · hourly timer] --> STATE

HCL shape (worker fills in; the plan output decides apply):

variable "test_fleet_ips" {
  description = "Pinned test-fleet addresses (today's leases, made static). env/test.env is DERIVED from these."
  type        = map(string)
  default = {
    memgraph = "10.53.2.10", redis = "10.53.2.11", prolog = "10.53.2.12",
    memgraph-ha-main = "10.53.2.20", memgraph-ha-replica = "10.53.2.21",
    rh-coord-1 = "10.53.2.30", rh-coord-2 = "10.53.2.31", rh-coord-3 = "10.53.2.32", rh-data-1 = "10.53.2.33",
  }
}

resource "incus_instance" "test" {
  for_each = local.test_fleet
  # … as today …
  device {
    name = "eth0"
    type = "nic"
    properties = {
      "name"         = "eth0"
      "network"      = incus_network.test.name
      "ipv4.address" = var.test_fleet_ips[each.key]
    }
  }
}

output "test_fleet_endpoints" {
  value = {
    TEST_MEMGRAPH_URI = "bolt://${var.test_fleet_ips["memgraph"]}:7687"
    MEMGRAPH_URI      = "bolt://${var.test_fleet_ips["memgraph"]}:7687"
    REDIS_HOST        = var.test_fleet_ips["redis"]
    REDIS_PORT        = "6379"
    CXP_GRPC_ENDPOINT = "${var.test_app_ip}:50053"
    # …
  }
}

Config ownership after WI-ENV: terraform → outputs → gen_env.py → env/*.env + .env.example → gate. The host .env (secrets, gitignored) keeps only secrets and the TEST_MEMGRAPH_URI line that tests/conftest.py reads; provision_worktree_env.py copies it. A commented-out owned key is a gate failure.

5. Swarm execution plan

Increment & verification protocol + hard rules

Inherited verbatim from plans/arch-eval-remediation-swarm-plan.md ("Increment & verification protocol", "Hard rules for every work item"): ordered parts, each with a ✓ Verify; red before green; one --no-verify commit per verified part; stop on ⛔; scope discipline; worktree per item via scripts/worktrees.py create; tests only via env -u PYTHONPATH .venv/bin/python scripts/run_tests_streaming.py <paths> -j 8 or run_tests.py <targets> --skip-gates --skip-infra --force; no mocks/stubs/fakes/monkeypatch; do not edit shared conftests; never delete; never merge/push; no AI attribution; final report with diffstat + verifying output + anything NOT done. Every brief names plans/contracts/eight-failures-2026-10-08.mmd (RUL-022). RUL-034 applies to every WI that touches src/: @traces, RTM regen, docs/plan status flip, wiki note, ≥90% coverage on touched files.

Phase 0 — central, before dispatch

• P0.1 Commit this plan + the contract on the integration branch feat/recorded-run-20261008-redesign (worktree wt-IRC-FLEET, based on origin/master 1d5f3e52c). ✓ git log --oneline -1.
• P0.2 Infra: start_test_services.py --status all up incl. the kernel. ✓ (hook banner at session start: Redis/Graph/Gateway/Kernel/Link running).
• P0.3 Mint worktrees off the integration branch: wt-ENV-DERIVE, wt-TOOLS-CLOSURE, wt-FLOSS-CORPUS, wt-TESTGEN-PATHS (tranche 1), wt-UNITS-PROBE (tranche 2). ✓ worktrees.py list shows each provisioned.
• P0.4 Slot leases: one contiguous range per worktree (scripts/slot_lease.py acquire), so concurrent suites never share orchkg_w<N>. ✓ slot_lease.py list.
• P0.5 Brief packs: scripts/gen_brief_pack.py --files … --symbols … per WI, written beside the brief. ✓ pack files exist.
• P0.6 Supervision: scripts/swarm_supervisor.py watch per worktree (stall alarm 1500s). ✓ status lists all.

Tranche 1 — parallel, disjoint writable scopes

WI-ENV (wt-ENV-DERIVE, agent: builder) — §3.1 / §4

• Files: infra/terraform/{test_fleet.tf,variables.tf,outputs.tf}, infra/gen_env.py, scripts/check_env_fleet_agreement.py (new), scripts/health_check.py (one check registration), deploy/ship.sh (verify step), env/test.env, env/dev.env, .env.example, tests/infra/test_gen_env_profiles.py (new), tests/scripts/test_check_env_fleet_agreement.py (new), tests/core/test_test_memgraph_routing.py (extend), plans/affected-tests-manifest.json (rules for the new files).
• Parts: 1 capture tofu output -json from the host into a fixture file (read-only) ✓ fixture holds test_fleet_ipv4 with the ten addresses · 2 red tests for EnvDeriver.env_lines(test) and the gate ✓ fail for the right reason · 3 implement TerraformOutputs/FleetProfile/EnvDeriver/FleetAgreementGate ✓ green; gen_env.py --profile test prints lines equal to env/test.env's owned keys · 4 HCL: pins + outputs; tofu validate + tofu plan -detailed-exitcode ✓ exit 2 with only in-place device adds and the new outputs; ⛔ any must be replaced · 5 wire the gate into health_check.py + ship.sh ✓ health_check.py lists it green on the current tree · 6 .env.example regenerated through --template ✓ routing test green.
• Done when: gate green on the integration tree, plan output attached to the report, no apply performed.

WI-DEPLOY (wt-TOOLS-CLOSURE, agent: builder) — §3.2

• Files: scripts/tools_import_closure.py (new), deploy/tools-vendor-manifest.txt (new), deploy/install-tools.sh (archive list + smoke), src/orchestrata_ultimate/refactor/codemod/persistence/__init__.py (new, docstring only), tests/scripts/test_tools_import_closure.py (new), tests/deploy/test_tools_vendor_manifest.py (new), plans/affected-tests-manifest.json.
• Parts: 1 red test: manifest-vs-closure mismatch on today's four subtrees ✓ fails naming verification, openspec · 2 implement the closure script ✓ --eager on _daemon_entry → 1 module; all-scopes with review_checks → 103 modules, six packages, matches §2.2 · 3 manifest + installer reads it ✓ bash -n; a --prefix /tmp/claude-… smoke install succeeds · 4 smoke constructs InMemoryReviewEngine() and reviews a 2-file workspace in the deployed venv ✓ passes on the smoke prefix; ⛔ if a third-party dep is missing, add it to the install line from the script's third_party output, re-run · 5 persistence/__init__.py ✓ find_spec resolves.
• Done when: smoke-prefix install proves a default review; manifest test green.

WI-FLOSS (wt-FLOSS-CORPUS, agent: builder) — §3.3

• Files: src/orchestrata_ultimate/core/tracing/scanner.py, scripts/generate_rtm_evidence.py, scripts/check_feature_loss.py, tests/core/tracing/test_scanner.py, tests/scripts/test_generate_rtm_evidence.py, tests/scripts/test_check_feature_loss.py.
• Parts: 1 profile run_check(REPO_ROOT) phase by phase ✓ numbers in the report (baseline) · 2 red tests: ModuleCorpus parses each module once (count via a recording filter), collect_from == collect_traces on a fixture tree ✓ fail · 3 implement corpus + collect_from + harvester corpus= seam + run_check single corpus ✓ green; docs/rtm-evidence.json regenerated byte-identical (365 ids) · 4 measure the canary alone and under a concurrent run_tests_streaming.py tests/scripts -j 8 ✓ < 60s loaded; 120s budget untouched · 5 if collect_current_implementation/find_surface_losses re-parse src/, apply the corpus there ✓ second measurement.
• Done when: identical output, measured < 60s under load, task 581 closable with numbers.

WI-TESTGEN (wt-TESTGEN-PATHS, agent: builder) — §3.4

• Files: src/orchestrata_tools/test_generator/testgen/types.py, enhanced_generator.py, jinja_gen.py, stub_generator.py, tests/plugins/git_sentinel.py, tests/generated_core/test_tracing___init___generated.py, tests/generated_recovery/test_verification_autofix_import_fixes.py, tests/generated_recovery/test_verification_autofix_engine.py, tests/test_orchestrata_tools_testgen_types_coverage.py, tests/plugins/test_git_sentinel.py, tests/test_generate_pytest_determinism.py.
• Parts: 1 red: generator given a workspace: str param emits a tmp_path-based value and the test signature takes tmp_path ✓ fails · 2 implement the strategy; all four generators consume it ✓ green; determinism test green · 3 regenerate the three generated modules ✓ diff shows only path literals; the three modules pass from an empty cwd (cd to a scratch dir) · 4 sentinel detector ✓ a module that creates <root>/zzz_probe/ fails with that name; nothing else changes · 5 run the full tests/generated_core + tests/generated_recovery ✓ green.
• Done when: no generated test can pass by writing into the checkout root; the sentinel would have caught test_value/.

Tranche 2 — after tranche 1 merges

WI-UNITS (wt-UNITS-PROBE, agent: wirer) — §3.5

• Files: src/orchestrata_ultimate/promotion/stages_units.py, tests/promotion/test_promotion_units.py, tests/promotion/test_stages_coverage.py.
• Parts: 1 red: daemon_units(tmp, probe=DirectoryUnitProbe(())) excludes a unit whose Requires= is unknown to the probe; with the dir containing it, includes it ✓ fails · 2 implement Protocol + two probes + default ✓ green; shipped U-UNIT2 test unchanged and green · 3 coverage ≥ 90% on stages_units.py ✓.

Central merge + final gate

1. Merge each WI branch into feat/recorded-run-20261008-redesign in dependency order (ENV, DEPLOY, FLOSS, TESTGEN, then UNITS); resolve nothing by hand that a re-run of the WI's verify would not prove.
2. scripts/affected_tests.py --since origin/master → run the proven set; then the RUL-032 full recorded run through deploy/ship.sh (verify → push → merge master → deploy ~/.claude + /opt/orchestrata-tools → promote_env.py test test → promote --to prod --yes), as a systemd-run --user unit, observed live.
3. Identity verification after promote: /opt/orchestrata/VERSION == origin/master, 14 units, /opt/orchestrata-tools smoke includes the default review.
4. Close tasks 580, 581, 575 with the measured numbers; file follow-ups for anything a WI reported as NOT done.

6. Decisions Mohamed owns (not taken by the swarm)

1. Apply the test-fleet pin (tofu apply of WI-ENV's plan): in-place device adds, addresses unchanged; Incus applies a nic ipv4.address on restart, so the nine containers restart once, one at a time, outside a recorded run.
2. git rm infra/terraform/memgraph_test.tf and var.memgraph_test_ip (the file header already says this is your command).
3. Stray test_value/ directories at orchestrata-ultimate, orchestrata-ultimate-architecture, wt-AST-DAEMON-P5, wt-FLEET-IMPORT, wt-IRC-FLEET — generated-test pollution; untracked; nothing references them. Deletion is yours.
4. Kimi / minimax bots remain the open IRC gap (not one of the eight): recovery needs the copy of the old ~/.claude you kept; otherwise rebuild at agent level per RUL-018 from wiki/SESSION_START.md.

7. RUL-034 Definition of Done per WI

8. Execution record (2026-10-08) — status: BUILT, merged on feat/recorded-run-20261008-redesign

Five workers (four builder, one wirer), one worktree each, supervisor watches + slot leases, dispatched against the contract above. Every WI verified centrally before its merge.

Cross-cutting findings

• scripts/run_tests_streaming.py numbers worker slots from 0 regardless of slot_lease.py; four concurrent swarms shared orchkg_w0..7 (task filed 2026-10-08; P0.4 of this plan is advisory until the runner honours SLOT_LEASE_BASE).
• env -u PYTHONPATH strips the shim that provides the injected orchestrata_pytest_observer plugin on this host; workers ran with PYTHONPATH=<wt>/scripts.
• Every AST-toolkit call regenerates plans/ast-*.json in the worktree; workers restored them before committing.
• The installer's "resolves into a working tree" guard matches the substring workspace, so the session scratchpad path cannot be a smoke prefix.

Decisions added for Mohamed (beyond §6): ADR-0078 (deployed daemon's Prolog review phase); after prod promote the hourly orchestrata-infra-drift.timer reports the nine pending nic pins as drift until the apply in §6.1 — ship.sh itself does not run the drift gate, so the ship is not blocked.

8.1 Central merge + final gate (2026-10-08, after the five merges)

affected_tests.py --since origin/master → 107 paths (escalate: false; reference-edges regenerated), expanded by the runner to 128 modules. First run at 8 workers beside three other swarms (load ≈27): 6 red. Each was investigated, not re-run blind:

Second run (4 workers, after e8ef57480): 126/128 green; the two reds are the rows above. Found on the way, both would have failed the real ship after its ten-minute suite:

• ship --dry-run wrote reports/.ship-verified (bare printf since 1f02eb894, 2026-08-19): a dry run stamped HEAD as the last verified tree without a test executing; the next real ship measures its blast radius FROM that commit — HEAD vs HEAD, empty set, smoke floor only — and would have certified 21 unverified commits. Fixed 5664022fc (write behind the DRY check, would stamp: announced) with a behavioural test that runs the real script in a scratch orchestrata repo (test_a_dry_run_leaves_the_verified_stamp_untouched); ship test modules 9/8/11/9/6 green. Working-tree stamp restored to the committed c4597f459, so the ship measures from a verified ancestor (22 commits).
• gen_plugin.py --check: 3 drifted items — WI-ENV's infra/terraform/{outputs,test_fleet,variables}.tf had no regenerated mirror under plugins/orchestrata. Regenerated, byte-identical to source (cmp), ab13a8032.

Cheap gate checks proven locally before the ship: check_env_fleet_agreement.py OK (test, dev match terraform + .env.example); gen_plugin.py --check current.

The recorded full run (RUL-032), before the ship. promote_env.py test test on a32b14194, as a user systemd unit (promote-test-112256) observed live. First launch aborted in 2s (aborted: freshness: the unit's minimal PATH had no swipl, so the freshness guard's reindex failed and said so only in the trace_intel lines — ADR-0079 item 4). Relaunched with the manager's environment:

The first ship (ship-eight-failures-113201) was refused by its own gate — correctly: run_tests.py --affected found two changed files that NOTHING tests, plans/contracts/eight-failures-2026-10-08.mmd (outside src/tests, no reference edge) and src/orchestrata_ultimate/refactor/codemod/persistence/__init__.py (zero test import coverage). A contract nobody checks against the tree is a drawing, so both got real tests rather than exclusions: tests/scripts/test_eight_failures_contract.py validates the diagram with the real validate_mermaid tool and asserts every class the built namespaces declare resolves to the symbol the worker realized it as (the table in the test records the recorded deviations: CitationHarvester → harvest_test_citations, FeatureLossGate → run_check, StagesUnits → daemon_units, VendorManifest → ClosureReport.vendor_manifest), with the recorded namespace the only unbuilt one; tests/refactor/codemod/test_persistence_package.py proves the package is regular and that the all-scopes closure of the daemon's graph writer resolves it (the deploy reason it exists). The diagram's non-import edge is declared in plans/affected-tests-manifest.json, like the SPEC-110 contract before it. Measured: contract 3/3, persistence 2/2, manifest rules 18/18.

The ship (ship-eight-failures-174429, 2026-10-08 17:44–18:10 EDT). Gate: ruff, black, RUL-007 clean; blast radius measured from c4597f45; the green ledger reused 124 modules proven in the recorded run and re-ran the 6 touched since — 3,233 passed, 0 failed; gen_plugin --check and the fleet-agreement gate OK; run evidence committed. Push → master fast-forwarded 29 commits to 6b7107ef7. Deploys, each identity-verified at 6b7107ef: ~/.claude (143 files, plugin cache 222 files), /opt/orchestrata-tools (333 files; both daemons answer the deployed commit; the vendored lib runs a default review — phases 1–2, phase 3 awaits ADR-0078), /opt/orchestrata-gitlab-runner. Promote prod: tree 1d5f3e52c6a4 → 6b7107ef7553, 6 services restarted, identity OK on 13 units, settled 11.0s, NRestarts 0, both graphs bootstrapped (1135s — the graphs stage dominates the promote). promote_env.py status ledger: prod promote=DEPLOYED sha=6b7107ef7553.

The ship exited 1 on its last check, as this plan predicted (§8, decisions): orchestrata-infra-drift.service fired at 18:00 from the promoted tree, whose terraform now declares the nine test-fleet nic pins, and reported Plan: 0 to add, 9 to change, 0 to destroy → the oneshot is in failed state, which ship.sh's 5b treats as a failed promotion. Every other stage passed; the product runs master. The unit will fail every hour until §6.1's tofu apply (Mohamed's call: nine in-place nic changes on the test fleet, no destroy) — the gate is doing exactly what ADR-0077 asked.

Found while preparing the promote: the prod interlock admits any latest green test record. Promoter.gate_for("prod") compares result only; the morning's test=PASS@10:27Z for 1d5f3e52c would have admitted this tree. Minted ADR-0079 (Proposed) — the test record carries the commit + source id and the gate requires the sha to match — docs 57b352758, wiki 9bb2f1d (shipped), graph synced, check_adr_graph_sync OK. Until ruled, the operator rule applied here: the recorded run above IS this tree's record, taken before the ship.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks