Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0075

ADR-0075: The promote's infra stage proves the ADR-0066 secrets file holds every required env key before the tree moves

← ADR-0074 · ADR-0076 →

The record

Status: Proposed (recommendation: Accept) · Date: 2026-10-07

Context

The prod promote runs an authoritative env preflight: promoter.validate (promotion/promoter.py:132-137) invokes scripts/validate_env.py with the profile overlay as the process env, so its exact REQUIRED_KEYS contract gates the promote. The infra stage then proves reachability — Redis, the graph cluster (promotion/env_profile.py:132 probe_infra; promoter.py:152 "infra is not reachable — refusing"). Neither step ever reads the ADR-0066 secrets file (~/.config/orchestrata/.env, mode 0600), and validate_env.REQUIRED_KEYS deliberately excludes the two CONVERSATION_KG_* keys (core/infra_env_rule.py's docstring: "The conversation keys are deliberately NOT in validate_env's REQUIRED_KEYS … validate_env is a RUNTIME PREFLIGHT (these keys must be present before…)"), so no promote stage proves the one file that gates the conversation consumer.

Measured cost of that gap, three times over (all before today's manual fix):

• P5 ship-2, 2026-10-06: promote ran to verify and failed — orchestrata-conversation-consumer.service: no process (MainPID=0), RuntimeError: CONVERSATION_KG_URI and CONVERSATION_KG_DATABASE not set (conversation_consumer.py:92-99).
• S2 attempt-3, 2026-10-06 ~21:35: verify FAILED after 90.4s — the consumer exited three times and systemd hit its start-rate limit (NRestarts=3); rollback's consumer restart failed identically.
• S3 attempt-2, 2026-10-07: rollback could not restart the consumer for the same reason; prod sat with the unit failed.

Every failure happened at the END of the pipeline: the tree had moved, the units had restarted, and the promote then rolled back and restarted them again. The detection was a three-line file read the promote never did — proven today, 2026-10-07, when the operator appended the two keys (named in ADR-0072, ruled Accepted) and the consumer started on the first try (orchestrata-kg-ready OK, daemon consuming).

ADR-0074 (Accepted 2026-10-07) fixed this exact failure shape for the embedding provider: probe at infra, refuse before the tree moves, no rollback because nothing moved. This ADR is the env-keys counterpart: the promote's own preflight must prove the secrets-file contract it ships against, not rediscover it at verify.

Decision (proposed)

1. The infra stage gains a required-env-keys probe, before the tree moves. It reads the ADR-0066 secrets file on the target host and checks every key in the required set is present and non-blank; a miss is [FAIL] infra env keys: missing CONVERSATION_KG_URI, CONVERSATION_KG_DATABASE and the promote is refused — no tree move, no restart, no rollback. The probe reads ~/.config/orchestrata/.env (the ADR-0066 source of truth), not /opt/orchestrata/.env, so a key added per a newly Accepted ADR is never refused merely because the installer's copy step has not run yet.
2. The required set is the infra_env_rule registry. That module already documents it as validate_env.REQUIRED_KEYS plus KG_URI plus the two CONVERSATION_KG_* keys; the probe imports that same set (no second list), and a test asserts the probe and the registry cannot drift.
3. Every addition is recorded by key name (ADR-0066's rule): the refusal message names the missing keys verbatim, and this ADR's register row records the CONVERSATION_KG_URI / CONVERSATION_KG_DATABASE pair as the keys the first enforcement names. The probe never writes the file and never proposes values (RUL-003: no literal infra defaults; secrets are operator-set).

Alternatives rejected

• Add CONVERSATION_KG_* to validate_env.REQUIRED_KEYS. Wrong surface: validate_env gates suite/checkout preflights against the process env, and dev/test sessions deliberately write nowhere (the keys stay optional there — infra_env_rule.py's recorded reason). The prod secrets file exists only on the target host, where the promote runs.
• Check at verify after the restarts. The status quo; measured cost is the three deploy → restart → 90s-fail → rollback cycles above.
• Auto-write the keys (or defaults) into the secrets file. Violates ADR-0066 ("grows only by keys the promote's stage proves missing — each addition recorded by key name") and RUL-003's no-literal-default rule.
• A manual operator checklist. Untestable, and the checker already exists in substance (scripts/check_env_contract.py; the promote can read a file in three lines).

Consequences

On Accepted: packet P6d of plans/ast-daemon-deployment-plan.md (or folded into P6c, beside ADR-0074's provider probe) adds the probe to promotion/env_profile.probe_infra (reusing the check_env_contract-style file parsing), with real-file tests on the test fleet proving: a missing key refuses at infra with the key's name and no unit restarted; a blank value counts as missing; a present pair passes and changes nothing; and the registry-drift test fails if infra_env_rule's set and the probe's set diverge. The R16b promote-failure class (consumer unit exit → start-rate limit → verify fail → rollback) becomes unreachable: the promote refuses in seconds, naming the missing keys. ADR-0069 applies (@traces, RTM, docs, wiki, ≥90% coverage, one ship).

Provenance

ADR-0066 (Accepted 2026-09-24) item R16b — the endpoint value "still to be named by Mohamed", named in ADR-0072 (Accepted 2026-10-07). Orchestrator finding F-P6a-3, 2026-10-07, from the three promote attempts above and the manual resolution (keys appended to ~/.config/orchestrata/.env and /opt/orchestrata/.env, consumer active (running) at 14:36 EDT). Code: promotion/promoter.py:132-152, promotion/env_profile.py:33-34, 132, core/infra_env_rule.py, scripts/validate_env.py:27, knowledge_graph/conversation_consumer.py:92-99. Awaiting Mohamed's ruling; only he rules.

Related ADRs

• Also mentioned: ADR-0066, ADR-0072, ADR-0074, ADR-0069

Rulings cited

• RUL-003: RUL-003 — Env config from the active env profile, read at call time — no hardcoding, no literal infra defaults

Specs cited

None (or no spec page exists for the ones named).