Orchestrata Ultimate Runbooks

Runbook operations at a glance

Start Infrastructure

python scripts/start_test_services.py

Run Tests

# Semantic graph subset
uv run pytest tests/semantic_graph/ -v --timeout=120

# Full suite (batched executor, NEVER xdist)
python scripts/run_tests.py

Check Memgraph (orchgraph Raft) Cluster

# Cluster state — SHOW INSTANCES via a Raft coordinator (routing 7691, no auth)
uv run python -c "
from orchestrata_ultimate.core.graph_factory import create_graph, NO_AUTH
d = create_graph(uri='bolt://10.53.1.160:7691', auth=NO_AUTH)
with d.session() as s:
    [print(dict(r)) for r in s.run('SHOW INSTANCES')]
"

Recreate / re-enroll the orchgraph cluster

The legacy incus/memgraph-cluster-launch.sh recreates the DELETED pre-Raft cluster — do not use it. Rebuild/re-enroll via infra/rehearsal/provision_node.sh + infra/rehearsal/enroll_orchgraph.py (see Reproducibility & IaC and the orchgraph-memgraph skill).

Prod user services — boot startup contract

Five long-running prod services run as user units in ~/.config/systemd/user/ (WantedBy=default.target), plus two timers:

unit	role	start ordering
`orchestrata-kernel.service`	Agent Runtime kernel (CXP over cxpt, `CXP_GRPC_PORT`)	kg-ready probe
`orchestrata-gateway.service`	WebSocket gateway (JSON-RPC 2.0, :8765)	`After=orchestrata-kernel.service`; kg-ready probe + 60s CXP-port wait (`-` prefix: falls back to in-process kernel by design if the port never opens)
`orchestrata-bot.service`	Orchestrata IRC channel bot	kg-ready probe; `Requires=miniircd.service`
`orchestrata-ast-live-sync.service`	file save → orchkg graph sync	kg-ready probe
`orchestrata-conversation-consumer.service`	SPEC-021 Redis queue → orchkg drainer	kg-ready probe
`orchestrata-history-mining.timer`	quarterly ruling-drift miner	oneshot
`orchestrata-kg-retention.timer`	weekly KG retention sweep	oneshot

The kg-ready probe is the load-bearing piece. ExecStartPre=/usr/bin/timeout 180 ~/.local/bin/orchestrata-kg-ready blocks until a READ-routed RETURN 1 succeeds against $KG_DATABASE. A bolt TCP-open check is NOT readiness: after host boot the orchgraph cluster accepts connections before READ routing is up, and services that started in that window either init degraded (kernel: "no audit trail", never retries) or crash (ReadServiceUnavailable) — the 2026-08-04 boot. Connection details come from the units' EnvironmentFile chain (.env prod-BASE + env/prod.env overlay, ORCHESTRATA_ENV_PROFILE=prod).

Every unit with the probe MUST set an explicit TimeoutStartSec (200s; gateway 260s for its second start-pre). cachyos-settings ships DefaultTimeoutStartSec=15s for the user manager, which otherwise SIGTERMs the probe mid-wait — this exact regression burned the 2026-08-11 boot after a unit rewrite dropped the setting (history in the cachyos runbooks). The units carry a comment saying so; do not strip it.

All five use Restart=on-failure + StartLimitIntervalSec=300/StartLimitBurst=5 (in [Unit], as systemd ≥230 requires) so a persistent construction fault surfaces in systemctl --user status instead of spinning. Env reload = systemctl --user restart <unit> — never hand-start these processes, a hand-started process snapshots .env forever (the 2026-07-26 cutover served graph traffic against the RETIRED Neo4j for ~17h that way).

Health check after any boot:

systemctl --user --failed                      # must be empty
journalctl -b -p err | grep 'Failed to start orchestrata'   # must be empty

KG orphan audit & connectivity census

# Authoritative: every node reachable from a :Project root (session-254 ruling).
# Native in-graph BFS since 2026-10-13 — ~47s on the 1.09M-node orchkg.
ORCHESTRATA_ENV_PROFILE=test python scripts/check_kg_connectivity.py

# Canonical-schema conformance (constraints + keyless-node scan)
ORCHESTRATA_ENV_PROFILE=test python scripts/kg_schema.py check

Full diagnosis + repair recipes (floating subtrees, unlinked Patterns, untagged test pollution, Tier-J strays): procedures/kg-orphan-detection-and-repair. 2026-10-13 incident: 28,889 orphan modules (incremental-sync never MERGEd the Project root) — repaired to 0 unreachable of 1,088,711; loader fix + regression tests in commit 7a29723dd, census rewrite in 196ce76a7.

Push to GitLab

unset GITLAB_TOKEN GITLAB_ACCESS_TOKEN OAUTH_TOKEN
git push origin master

Reproducibility & IaC

See Reproducibility & IaC for the full picture.

# Canonical-schema conformance of the live operational KG (must exit 0)
python scripts/bootstrap_graphs.py --kg --check

# One-shot canonical migration (idempotent) — dry-run first, back up before writing
python scripts/canonical_conformance.py --dry-run
python scripts/canonical_conformance.py

# Infra drift gate — live fleet vs Terraform (exit 2 = drift)
terraform -chdir=infra/terraform plan -detailed-exitcode

# Rebuild rehearsal from an empty isolated network (see infra/rehearsal/README.md)
cd infra/rehearsal && ./run_rehearsal.sh apply   # ... teardown when done