Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › ph1-prod-unit-health-builder

PH1 — prod unit health: builder brief (orchestrator's picks, 2026-10-07)

Source: plans/briefs/ph1-prod-unit-health-builder.md at 8a2cb42ad · last changed 2026-10-07 · first committed 2026-10-07 · 90 lines.

Prose pending.

Plan text

PH1 — prod unit health: builder brief (orchestrator's picks, 2026-10-07)

Plan: plans/ast-daemon-deployment-plan.md § "PH1 — prod unit health". Read that section first; it carries the diagnosis. This brief carries the exact source, the change and the proof.

1. The defect, measured

Over the eleven templates in deploy/systemd/, six gate on orchestrata-kg-ready.py in ExecStartPre; five of those declare Environment=ORCHESTRATA_ENV_PROFILE=prod and EnvironmentFile=-/opt/orchestrata/env/prod.env; orchestrata-kg-retention.service does not. /opt/orchestrata/.env is secrets-only (ADR-0066), so deploy/orchestrata-kg-ready.py (profile = source.get("ORCHESTRATA_ENV_PROFILE", "test"), line 203) resolves the TEST profile and the prod oneshot waits 120 s on bolt://10.53.2.10:7687, then fails in ExecStartPre. Journal lines of the installed unit, verbatim:

orchestrata-kg-ready: waiting up to 120s for bolt://10.53.2.10:7687 (db=orchkg)
orchestrata-kg-ready: bolt://10.53.2.10:7687 did not answer within 120s after 61 attempt(s)
orchestrata-kg-retention.service: Control process exited, code=exited, status=1/FAILURE

2. Exact source at e50693b9e

deploy/systemd/orchestrata-kg-retention.service (whole file):

[Unit]
Description=Orchestrata KG retention — age out gateway audit exhaust (30d) + CXP exhaust sweep
After=network-online.target

[Service]
Type=oneshot
WorkingDirectory=/opt/orchestrata
EnvironmentFile=/opt/orchestrata/.env
# Boot catch-up guard (same rule as every KG-dependent unit): block until a
# READ-routed `RETURN 1` succeeds — a bolt TCP-open check passes before READ
# routing is up (2026-08-04 boot).
ExecStartPre=/usr/bin/timeout 180 /opt/orchestrata/.venv/bin/python /opt/orchestrata/deploy/orchestrata-kg-ready.py
ExecStart=/opt/orchestrata/.venv/bin/python scripts/kg_retention.py --retention-days 30 --cxp-exhaust --apply

deploy/systemd/orchgraph-backup.service lines 8-15 (the shape to copy, comment included):

EnvironmentFile=/opt/orchestrata/.env
# Explicit prod profile overlay, as on the long-running units. Since ADR-0066
# .env is secrets-only, so without this the readiness probe below resolved NO
# endpoint from the environment and fell back to the TEST profile: it gated a
# PROD backup on the test graph (measured 2026-10-05 — every run waited 120s on
# bolt://10.53.2.10:7687 and failed while the prod cluster was healthy).
Environment=ORCHESTRATA_ENV_PROFILE=prod
EnvironmentFile=-/opt/orchestrata/env/prod.env

tests/deploy/test_deploy_assets.py: module constants REPO_ROOT, DEPLOY, SERVICE_UNITS (lines 44-59; SERVICE_UNITS globs deploy/systemd/*.service and deploy/orchestrata-test/*.service), helpers parse_unit(unit) -> dict[str, list[tuple[str, str]]], _keys(section), _rel(path); the parametrized test test_service_unit_is_structurally_a_service (lines 137-151) is the shape to follow. Read them with .venv/bin/python scripts/serialize_codebase_ast.py source deploy.test_deploy_assets --residue --no-daemon and ... source deploy.test_deploy_assets.test_service_unit_is_structurally_a_service --no-daemon.

3. The changes

  1. Template. In deploy/systemd/orchestrata-kg-retention.service, directly after EnvironmentFile=/opt/orchestrata/.env, insert the backup unit's comment block adapted to this unit (say "it gated a PROD retention sweep on the test graph (measured 2026-10-07 — every timer run waited 120s on bolt://10.53.2.10:7687 in ExecStartPre and failed while the prod cluster was healthy)") followed by the two lines Environment=ORCHESTRATA_ENV_PROFILE=prod and EnvironmentFile=-/opt/orchestrata/env/prod.env. Nothing else in the file changes.
  2. Proof test in tests/deploy/test_deploy_assets.py, parametrized over SERVICE_UNITS like the existing one, named test_a_kg_gated_unit_declares_the_prod_profile: parse the unit; if no ExecStartPre value contains orchestrata-kg-ready.py, return (the test is about kg-gated units only; do not skip, return); otherwise assert ("Environment", "ORCHESTRATA_ENV_PROFILE=prod") is among the [Service] pairs and ("EnvironmentFile", "-/opt/orchestrata/env/prod.env") too, with _rel(unit) in the message. Follow the module's own decorator convention (check whether its tests carry @traces; if they do, cite only ids the deploy spec defines; if they do not, add none).
  3. Red first. Run the new test against the UNCHANGED template first and show it failing on deploy/systemd/orchestrata-kg-retention.service only; then apply the template change and show it passing on all units. Both runs go in the report with their log paths.

4. Gates and done-when