Orchestrata Ultimate / Deployment / Host services

Orchestrata host services: what each one is for

First written 2026-10-22; the register below was added 2026-10-23. Requested by Mohamed, 2026-10-22: "find out why the services were built, what purpose they served and document that". Extended on 2026-10-23 to "every module and service".

This page is the runtime view. It sits beside component-dag.md, the code view, and architecture-ontology.md, the design-knowledge schema.

ADR-0123 (Proposed, 2026-10-22): every host service is registered in the runtime view with its purpose and architecture component, or it is retired. The decision text reads:

• "the wiki page projects/orchestrata-ultimate/runtime-services.md is the register of host services."
• "A deploy never enables or starts a unit; enablement is the operator's."
• "A unit without a purpose and an architecture home is retired."
• "Each unit's disposition (keep enabled, on demand, or retire) is ruled one at a time."

A unit with no row in the register below has no standing. A row that says "not in the architecture docs" is a candidate for retirement under that rule, until it is ruled.

DeepChat disabled 16 units on 2026-10-21, and they are still disabled

On 2026-10-21 at 19:28, a DeepChat session followed Mohamed's instruction: "stop the production cluster, then investigate the user services and daemons and stop them, and the mcp servers and make them not start automatically". It reported that it had stopped and disabled 16 user units:

• 5 MCP servers: architecture-mcp, dev-tools-mcp, knowledge-memory-mcp, openmandriva-mcp, task-mcp.
• 11 user daemons: orchestrata-ast-live-sync, orchestrata-bot, orchestrata-canvas, orchestrata-conversation-consumer, orchestrata-gateway, orchestrata-kernel, miniircd, hermes-gateway, gitlab-runner, memory-monitor, aw-proxy.

It also stopped the system gitlab-runner and the prod orchgraph cluster. The source is the DeepChat session export, turn 204.

On 2026-10-23 at 00:35, systemctl --user is-enabled reports all 16 as disabled. Two of them are running anyway:

• orchestrata-conversation-consumer, started by hand on 2026-10-22 at 15:39 at Mohamed's request;
• the user gitlab-runner. Who started it was not measured.

The system gitlab-runner and the prod cluster are running again.

The 2026-10-22 15:31 record in Decisions on 2026-10-22 names six services: miniircd, bot, kernel, gateway, ast-live-sync and conversation-consumer. All six are in DeepChat's list, so they were already disabled on 2026-10-21. The only item in that record that is not in DeepChat's list is orchestrata-infra-drift.timer.

Register: every host unit (measured 2026-10-23 00:35 EDT)

Sources: units-roles.txt and services-census.txt (session scratch, 2026-10-22 23:30); the unit files in ~/.config/systemd/user/; and systemctl --user is-enabled|is-active, re-run on 2026-10-23. State is enabled-state/active-state. "static" means the unit has no [Install] section; it is pulled in by its timer or started by hand.

Orchestrata product units (run from /opt/orchestrata, profile env/prod.env)

Orchestrata tool units (run from /opt/orchestrata-tools)

The five running ast-daemon@ instances serve these worktrees: orchestrata-wt-unshipped-scope, wt-COMPONENTS, wt-INT-LAND, wt-MAP-PRECOMMIT and wt-TOOLS-FAST. install-tools.sh restarts every active instance after a tools promote.

Other orchestrata-named units

Units outside orchestrata

Desktop session units. The census also lists the desktop session's own user units. Six of them were enabled and active: fix-chromium-mic, gnome-keyring-daemon (with its socket), thumbnail-cleanup, x1-yoga-rotate, drkonqi-coredump-launcher.socket and drkonqi-coredump-cleanup.timer. flatpak-session-helper was static and active. They are part of the desktop, not of any project, and are out of scope for ADR-0123.

MCP servers Claude launches per session (stdio, not units)

How the services connect

The edges come from the unit files, ADR-0065, ADR-0099, AGENTS.md Hard Rule 4, and infra/README.md, which says prometheus "scrapes the gateway /metrics". The per-unit sections below cite each edge.

How this page was built

• Unit files and history: the worktree /home/mohamed/workspace/orchestrata-wt-unshipped-scope at 4ac0db570 (= origin/master = the commit prod was re-installed at on 2026-10-22). The main checkout is 497 commits behind and was not used for content. Unit files in deploy/systemd/ were compared byte for byte between the two; the ten units are identical.
• Code: read only through the AST toolkit (serialize_codebase_ast.py source|symbols|refs|who-imports, --residue for module docstrings). No .py file was read by hand. Where a claim is about code, the symbol and file are named.
• Decisions: docs/adr/, specs/, plans/, docs/, INFRASTRUCTURE.md, SESSION_START.md, AGENTS.md in the worktree; the knowledge-wiki (rulings/, projects/orchestrata-ultimate/, reference/, agents/).
• Live state: systemctl --user show|cat|list-*, journalctl --user, ss, ps. Reads only. No unit was started, stopped, enabled or disabled.
• Where nothing was found, the text says "no recorded rationale" or "unknown". Quotes are verbatim.

Architecture column: the two architecture pages component-dag.md (ten components, packages refactor) and architecture-ontology.md (design-knowledge schema) were read. architecture-ontology.md is a graph schema for design knowledge (Component, ADR, Requirement ...); it names no host service, so it maps no unit. component-dag.md lists the ten components (contracts, governance, knowledge_graph, kernel, session, volks, gateway, tools, integration, plugin). Its cut-over status: "Cut-over has not happened", so the units still launch orchestrata_ultimate.* modules, not the packages/ trees. Mapping below is therefore from the code location plus the per-component design census (packages/*/design/boundary-census.md), and says "not in the architecture docs" where neither page nor census covers the unit.

Summary

1. miniircd.service

What it runs. ExecStart=/opt/orchestrata/.venv/bin/python %h/.local/bin/miniircd --ports 6667 --listen 0.0.0.0 --state-dir %S/miniircd --channel-log-dir %S/miniircd/channels --verbose. No WorkingDirectory, no env file. miniircd is the upstream single-file server (2.3), executed, not imported. Unit description: "the local IRC server the agent fleet meets on (#agents, #orchestrata)" (deploy/systemd/miniircd.service). No Python entry point is involved, so there is no AST evidence to cite.

Why it was built. Commit 1d9ca9fda, 2026-10-08, "Own the IRC server: miniircd.service, and the listener the hook actually ran". Quote from the commit message: "miniircd had no unit. SESSION_START.md and orchestrata-bot.service's Requires=miniircd.service both named one; none existed. The server on :6667 was a nohup orphan of the SessionStart hook, living in whatever Konsole cgroup ran that session". Consequence stated there: "the bot was 'not runnable here' on every host and ship.sh skipped it on exactly that ground." There is no ADR, SPEC or ruling for why the fleet uses IRC at all. Where IRC is described: SESSION_START.md "IRC: Connect to #agents"; wiki reference/agent-tooling.md §7: "IRC remains the channel for live task dispatch the user watches." (Why IRC was chosen over another bus: no recorded rationale.)

What uses it.

• orchestrata-bot.service: Requires=miniircd.service and After= (unit file).
• The Claude IRC listener scripts/_irc_listen.py, started by the SessionStart hook claude/hooks/irc_agents_hook.sh (commit 1d9ca9fda). Measured 15:47 today: pid 11265, /opt/orchestrata/.venv/bin/python /opt/orchestrata/scripts/_irc_listen.py, up 17.5 h, 0.0% CPU, 12.8 MB, and ~/.local/state/irc-agents/claude-listener.log ends in repeated "[error] connect failed: [Errno 111] Connection refused; retry in 5.0s".
• The SPEC-021 IRC conversation producer: plans/spec-021-multiplatform-producers.md §2 extends _irc_listen.py/_irc_send.py and its tests "run against the real localhost:6667 miniircd".
• The hook: irc_agents_hook.sh lines 50-53: if systemctl --user cat miniircd.service succeeds it runs systemctl --user start miniircd.service (start, not enable); otherwise it falls back to nohup ... miniircd.
• The kimi/minimax bots described in SESSION_START.md (kimi-irc-bot.service) are not units on this host: commit 1d9ca9fda says they "do not exist on this host at all", and systemctl --user list-units shows none.

What breaks while it is off. The bot cannot run (Requires=). The Claude listener keeps retrying every 5 s against a closed port (measured above); nothing is received or sent on #agents. deploy/ship.sh lines 763-764 skip the bot as "not runnable here" when miniircd is unloadable. Other effects on agent-to-agent traffic: the wiki says kimi-session and agent-mail are the sync/async channels and "both off IRC" (reference/agent-tooling.md §7), so those are unaffected. Whether any other tool depends on #agents is not established.

Resources. None recorded for miniircd itself. The retrying listener is the only observable cost (0.0% CPU, a 2 MB log).

Architecture. Not in the architecture docs (searched component-dag.md, architecture-ontology.md, architecture.md: no match).

2. orchestrata-bot.service

What it runs. WorkingDirectory=/opt/orchestrata, EnvironmentFile=/opt/orchestrata/.env plus env/prod.env, ExecStartPre = deploy/orchestrata-kg-ready.py --timeout 215, ExecStart=/opt/orchestrata/.venv/bin/python /opt/orchestrata/deploy/orchestrata-bot.py. Code (toolkit --residue and source main on deploy/orchestrata-bot.py): module docstring "Orchestrata IRC channel bot - keeps #orchestrata channel alive." It connects to 127.0.0.1:6667 as orchestrata-bot, joins #orchestrata, answers PING, logs channel messages, rejoins when kicked, reconnects with backoff up to 300 s. It has four functions (irc_send, handle_ping, connect_and_join, main) and no command handling. It does not use the graph, although the unit gates its start on a graph probe.

Why it was built. no recorded rationale for the bot. Looked in: docs/adr/ (grep irc), specs/, plans/ (KIMI-TASK-flakes-and-unit.md, system-analysis-master.md), SESSION_START.md, wiki rulings/, agents/, reference/. What is recorded is only why the unit file is in the repo: commit 6e1e68f99, 2026-08-10, "The IRC bot unit ran only at ~/.config/systemd/user/orchestrata-bot.service with no copy under deploy/systemd/, so install.sh could not install it on a fresh machine and the unit-drift gate could not see it." It was first excluded on 2026-07-26 (83a3d5fd2: "orchestrata-bot.service runs a script outside this repo"). Commit 90bb03a61, 2026-08-14 moved the script into deploy/: "its ExecStart ran an unversioned ~/.local/bin/orchestrata-bot.py whose env-python3 shebang resolved to /usr/bin/python3.14". SESSION_START.md says it is "the same pattern" as the kimi bot ("nick orchestrata-bot in #orchestrata"). The oldest trace of the name in git is 2026-07-24 (f7c2b1d12, session state).

What uses it. Nothing found. Searches: git grep orchestrata-bot (non-Python): only SESSION_START.md ("Other agents in channel: claude, codewhale, orchestrata-bot (#orchestrata)"), the provision skill's status list, deploy/ship.sh, deploy/systemd/README.md. #orchestrata is named in no other file than the bot itself and SESSION_START.md. Dependency direction: it depends on miniircd.service.

What breaks while it is off. Nothing found beyond #orchestrata having no resident nick. No consumer found.

Resources. None recorded. The ExecStartPre graph probe can block up to 215 s (commit 0d772955a: "4 of 5 cold boots exceeded 120s").

Architecture. Not in the architecture docs.

3. orchestrata-kernel.service

What it runs. WorkingDirectory=/opt/orchestrata, .env + env/prod.env, ExecStartPre graph probe (kg-ready, 215 s) and mkdir -p /var/lib/orchestrata/session-root, ExecStart=/opt/orchestrata/.venv/bin/python -m orchestrata_ultimate.agents.runtime_main. Code (toolkit, runtime_main.py, _serve): resolves CXP_GRPC_PORT from the environment ("there is no hardcoded default"), builds a CXPServerConfig(host="0.0.0.0", port), builds the kernel packet handler (agents.kernel_handler.build_kernel_packet_handler), creates the CXP server, installs SIGTERM/SIGINT handlers, serves, and flushes the audit trail on exit. Its module docstring is stale: "Agent Runtime gRPC server entry point ... Used by Docker container orchestrata-agent-runtime"; the unit runs it on the host over cxpt (grpc retired 2026-08-07, 11ebfc2e7).

Why it was built.

• The unit: commit 45d2c83d5, 2026-07-27, "Gateway + kernel become systemd units - a hand-started daemon kept the old graph". The kernel-specific reason is in the unit file: "a hand-started kernel keeps whatever .env it was launched with. The kernel is REQUIRED infra - a suite run that does not start and CXP-verify it is invalid - so it must not be the one process nobody can restart reproducibly."
• The requirement: docs/kernel-is-required.md (the unit's Documentation=), added by commit 9a5ef5a94, 2026-07-12 ("feat(prolog): vendor free-threaded janus + make the kernel required, purge xdist"): "every authoritative test-suite run MUST start and CXP-verify the Agent Runtime kernel. A run that skips the kernel is not a valid run" and "If the kernel is not run and tested, the project fails." AGENTS.md Hard Rule 7: "The kernel is the beating heart - REQUIRED for every suite run."
• What the kernel is: docs/kernel-is-required.md: "the central coordination process of the system: agents, CXP dispatch, governance, and the pipeline all route through it."
• Prod placement: design census packages/volks/design/boundary-census.md N-V7 records that agents.runtime_main is "LAUNCHED BY THE KERNEL SERVICE: deploy/systemd/orchestrata-kernel.service:43".
• Earlier unit history: 98c401fd0 2026-08-07 (RUL-027), d532c38b9 2026-09-23 (ADR-0062: every prod unit runs from /opt/orchestrata).

What uses it. Toolkit refs create_cxp_client (src tree): definitions in cxp/transport.py:126, used by gateway/server.py:613 and kernel/dispatcher.py:691. So the gateway is the client on this host. AGENTS.md Hard Rule 4: "Gateway connects via the cxpt direct transport to kernel on port 50053 ... Falls back to PipelineKernelClient only when unreachable" (PipelineKernelClient is defined in orchestrator/pipeline_kernel.py:83, used by gateway/server.py:797). The gateway unit orders After= and Wants=orchestrata-kernel.service and waits up to 60 s for the port. INFRASTRUCTURE.md: the test fleet runs its own kernel in the app container; "The host --user units (deploy/systemd/) target prod".

What breaks while it is off. Per AGENTS.md Hard Rule 4 and the gateway unit comment: the gateway "falls back by design" to the in-process PipelineKernelClient ("a production resilience path"), so gateway calls keep being answered but not by the out-of-process kernel. The unit comment says the fallback "is meant for a kernel that is DOWN". What functions are lost in the fallback is not established here. For test runs on this host: AGENTS.md says a run without the kernel is non-authoritative; the authoritative test kernel is in the container, not this unit.

Resources. Commit 0d772955a, 2026-10-13: the kg-ready start budget was raised to 215 s "(4 of 5 cold boots exceeded 120s)". Wiki projects/orchestrata-ultimate/kernel-fd-leak-and-env-ownership.md exists (fd leak) but was not read for this page. No CPU figure for the kernel was found.

Architecture. Design census packages/volks/design/boundary-census.md N-V7 and decisions D-V21: agents.runtime_main "is a VOLKS module by grammar ... it COMPOSES kernel + volks"; "AD-25/D-K8 precedent puts PROCESS ENTRY POINTS in integration"; the unit string re-points at cut-over (packages/integration/design/boundary-census.md row: orchestrata_volks.agents.runtime_main <PENDING volks>). component-dag.md lists kernel (landed) and volks (landed); the unit is described as a "plugin-owned unit" (deploy/ AD-5).

4. orchestrata-gateway.service

What it runs. WorkingDirectory=/opt/orchestrata, .env + env/prod.env, ExecStartPre graph probe, then a 60 s tolerant wait for the kernel's CXP_GRPC_PORT, ExecStart=/opt/orchestrata/.venv/bin/python -m orchestrata_ultimate.gateway.cli serve. Wants= and After=orchestrata-kernel.service. Code (toolkit, gateway/cli.py): main loads .env, configures logging, and for serve calls run_serve, which builds GatewayConfig, starts WebSocketGateway, and blocks until cancelled. @traces spec 014 (REQ-014-101/102/166/170).

Why it was built.

• The gateway itself: SPEC-014 (specs/014-websocket-gateway/spec.md, created 2026-03-27). Quote: "The WebSocket Gateway is the real-time control plane for Orchestrata. It exposes a bidirectional WebSocket server that allows external clients (web dashboards, CLI tools, mobile apps, external integrations) to observe and control the orchestration system in real-time."
• The unit: commit 45d2c83d5, 2026-07-27. Quote (unit file, "The reason this unit exists"): "The gateway used to be hand-started with uv run python -m orchestrata_ultimate.gateway.cli serve, which snapshots .env into the process at launch and never revisits it. After the 2026-07-26 orchgraph cutover the running process still carried KG_ENGINE=neo4j and NEO4J_URI=... - it served graph traffic against the RETIRED Neo4j for ~17 hours, and start_test_services.py reused it, so the suite's gateway paths were green against the wrong database." Commit message adds: "JWT_SECRET existed only in an interactive shell, never in .env, so the gateway failed closed (SPEC-099) under systemd."
• Later: 5fe467e4b 2026-08-16 (prod fleet moved to 10.53.1.0/24); f2c8867b9 2026-08-16 (unit owns its start budget); ADR-0065 (Accepted, ruling R15, 2026-09-24: "1, proceed with that") pins GATEWAY_HOST=127.0.0.1 for the prod WS plane.

What uses it.

• ADR-0065: "Nothing on the host needs the prod WS control plane on a routable address: every client of it (the CLI, hooks, the kernel) runs on this host."
• orchestrata-mcp (the orchestrata MCP server, /opt/orchestrata/bin): ADR-0099 (Accepted 2026-10-13): "a WebSocket JSON-RPC client of the running gateway ... opens one WebSocket to the deployed gateway at GATEWAY_HOST:GATEWAY_PORT, authenticates with GATEWAY_TOKEN". Every mcp__orchestrata__* tool goes through it.
• The VizInt UI: AGENTS.md: "WebSocketBridge connects to gateway at ws://localhost:8765."
• The agents-packets CLI (context.history.md: "agents-packets CLI needs a live gateway").
• Port table: AGENTS.md ("Gateway | 8765 | WebSocket | JSON-RPC 2.0"), INFRASTRUCTURE.md.
• Toolkit refs WebSocketGateway: used only by gateway/cli.py:64 and gateway/server.py:1459.

What breaks while it is off. The MCP server has nothing to connect to. Coordinator-measured today: gateway_ping answers "GATEWAY_UNAVAILABLE: GATEWAY_TOKEN is not set", so on this host the MCP path is already unusable for a second, independent reason (no token), and the stopped gateway would be the third. VizInt UI and the CLI/hooks that use the WebSocket lose their endpoint. Gateway audit events stop being written (the writer is gateway-side, docs/gateway-audit-rollup-2026-07-26.md), which also removes the largest source of graph exhaust that kg-retention cleans. What exactly each hook does when the gateway is down was not traced.

Resources. Gateway audit volume was the largest bloat in the operational KG: "325,366 GatewayAuditEvent nodes + ~250K UUID-keyed connection Session nodes - ~72% of the operational KG" (rollup doc, "overwhelmingly test-suite traffic"). Start budget TimeoutStartSec=300 (kg-ready 215 s + kernel wait 60 s).

Architecture. Component gateway (orchestrata_gateway, landed in component-dag.md; packages/gateway/src/orchestrata_gateway/cli.py). Census row: "python -m orchestrata_ultimate.gateway.cli serve | systemd units (plugin, deploy/ AD-5)" (packages/gateway/design/boundary-census.md:150).

5. orchestrata-ast-live-sync.service

What it runs. WorkingDirectory=/opt/orchestrata, .env, env/prod.env, and env/ast-roots.env (the declared AST roots; the unit comment: "Without it the service has NO roots and refuses to start -- deliberately"). ExecStartPre graph probe, ExecStart=/opt/orchestrata/.venv/bin/python /opt/orchestrata/scripts/ast_live_sync.py, TimeoutStartSec=240. Code (toolkit --residue, module docstring of scripts/ast_live_sync.py): "Live code-intelligence sync: file save -> graph updated in seconds. Step 3.4 of plans/neo4j-to-orchgraph-migration-plan.md." Pipeline: watchfiles.awatch (debounced) -> orchestrata_ast.load_or_analyze (re-parses only files whose content hash changed) -> GraphSyncer.incremental_sync. "SLO: file save -> graph updated, end to end, < 5 s." "Deployed as the orchestrata-ast-live-sync user systemd unit -- always on, not a remembered command."

Why it was built. plans/neo4j-to-orchgraph-migration-plan.md Step 3.4: "Live code intelligence (session-254 requirement: sub-ms review queries, seconds-fresh sync)". Executed 2026-07-26 in commit 611300335 ("Step 3.4: live code intelligence - all five SLOs met and measured", file-save -> graph 1.31-1.68 s against a < 5 s target). That commit also added the unit: "orchestrata-ast-live-sync user unit (enabled + active, bolt-port readiness guard matching orchgraph-backup)" and "The unit is tracked in deploy/systemd/ so a rebuild restores it". Later changes: b0138e9cb 2026-10-09 (declared root set, because the former default "was how the graph lost every scripts/, claude/ and tests/ module"); d532c38b9 (ADR-0062).

What uses it. It writes the code-intelligence subgraph of orchkg. INFRASTRUCTURE.md: orchkg "Runtime + code intelligence ... kept current on every file save by orchestrata-ast-live-sync.service". Readers named there: scripts/graph_query.py, scripts/graph_browser.py; also the /code skill (orchgraph search), and the review and impact queries the SLOs were set for. Wiki prolog-is-not-self-learning.md §1 lists it as "Rebuilt - wired, deployed, running". Not the only path: the PostToolUse hook claude/hooks/ast_sync_hook.py (toolkit source main) runs /opt/orchestrata-tools/bin/ast_live_sync ... --once detached after a Claude Write/Edit/MultiEdit of a .py file inside a project's source package; its docstring: "keep the code-intelligence graph current, in ANY project". deploy/install-tools.sh and commit f39d5160a (2026-08-13) made the same script project-generic.

What breaks while it is off. The service stops syncing saves made by anything other than a Claude edit hook: git operations, edits by other agents (kimi/hermes/pi workers), formatter passes, edits to tests/, scripts/, claude/ (the hook returns early for files outside the project's source package: "not a source file of this project (tests, scripts, docs)"). Claude's own edits to src/ still sync through the hook as long as /opt/orchestrata-tools/bin/ast_live_sync exists. The graph therefore goes stale for everything else; the prod graph is only as fresh as the last sync. The prod re-install on 2026-10-22 changed the tree under /opt/orchestrata without a live sync.

Resources. Wiki prolog-is-not-self-learning.md recorded "12h uptime, 17.6s CPU" in the early days. No CPU incident recorded for this unit specifically. Related, for the sibling dev daemons (not in scope): orchestrata-ast-daemon@ "running with 1.7GB RSS" (commit 6f6714caf). Mohamed's 15:31 stop was to cut host CPU, per coordinator.

Architecture. The code writes through knowledge_graph.sync.GraphSyncer and orchestrata_ast (the latter lives in packages/knowledge_graph/src/orchestrata_kg/orchestrata_ast/), i.e. component knowledge_graph. ADR-0118 §5 lists ast_live_sync among "the deployed-from-here family (observe_pytest, swarm_supervisor, ship, ast_daemon, ast_live_sync) -> one-deploy-lane question each get their own ruling" - undecided.

6. orchestrata-conversation-consumer.service

What it runs. WorkingDirectory=/opt/orchestrata, .env, env/prod.env, ExecStartPre graph probe, ExecStart=/opt/orchestrata/.venv/bin/python /opt/orchestrata/scripts/run_conversation_consumer.py. Unit description: "orchestrata SPEC-021 conversation-turn consumer (Redis queue -> orchkg graph)". Code (toolkit --residue, docstring of scripts/run_conversation_consumer.py): "Standing SPEC-021 conversation-turn consumer daemon." It loops ConversationQueueConsumer.flush() (claim -> parse -> map -> write -> ack; never raises, REQ-021-070), fast-loops while there is work, sleeps --interval (default 5 s) when idle, logs totals every 300 s, handles SIGTERM with a final flush. Queue key default orchestrata:conversation:queue.

Why it was built. Commit 90883df0f, 2026-08-04, "Add standing conversation-turn consumer daemon (SPEC-021 automatic sync)". Quote: "The SPEC-021 turn-capture producer (user-level Claude Code hook) writes one message per conversation turn into the Redis reliable queue orchestrata:conversation:queue on every turn. Nothing drained that queue on a standing cadence: the kernel only flushes it at boot and shutdown ..., so between kernel bounces turns piled up unsynced (6078 backlogged in prod at the time of this change)." Design reason (script docstring): "Decoupled from the kernel on purpose: turns are captured on every Claude turn independent of kernel uptime, so the drainer must run independently and survive kernel restarts (same rationale as scripts/ast_live_sync.py)." Specs: SPEC-021 US-021-007 (cited by ADR-0072); producers plan plans/spec-021-multiplatform-producers.md. Endpoint decision: ADR-0072 (Accepted, Mohamed, 2026-10-07: "cluster = Memgraph, DB = orchkg (ADR stands)"). Note: specs/021-opencode-integration/spec.md itself is the OpenCode integration spec; the turn-capture requirements (REQ-021-069/070) are cited in code and ADRs, and I did not locate their text there.

What uses it (what feeds it, what reads it).

• Producers into the queue: the user-level Claude hook ~/.claude/tools/claude_turn_capture.py (script scripts/claude_turn_capture.py), and the kimi/IRC producer (plans/spec-021-multiplatform-producers.md §2). Producers "exit 0 and append one line per drop to $XDG_STATE_HOME/orchestrata/conversation_capture.log" (deploy/systemd/README.md). The tail of that log shows "TimeoutError: Timeout connecting to server" at 2026-10-22 03:56 UTC.
• Readers of what it writes: /remember -> ~/.claude/tools/conversation_recall.py (claude/commands/remember.md: turns are "queued through Redis, and ingested into the prod conversation graph by conversation_consumer"); scripts/mine_prompt_history.py (history-mining, primary input).
• Toolkit who-imports orchestrata_ultimate.knowledge_graph.conversation_consumer: kernel/cli_main.py:606 (_boot_conversation_consumer, gated by ORCHESTRATA_CONVERSATION_CONSUMER) and tools/conversation_tools.py:32. The unit's kernel entry agents/runtime_main.py is not a direct importer; whether it reaches cli_main indirectly was not traced. So, from direct evidence, this unit is the only standing drainer among the ten.
• Unit ordering: none (After=network-online.target).

What breaks while it is off. Turns are not lost: they accumulate in the Redis list (6078 backlogged before this unit existed; commit 90883df0f drained all of them with 0 failed). Until drained, /remember cannot find recent turns and the history miner sees a stale corpus. Today: started 15:39 at Mohamed's request, it flushed 51 queued turns (coordinator), and the journal shows continuing one- and two-turn flushes (e.g. 15:44:25 "Flushed 2 conversation turns to graph (0 quarantined, 0 failed)").

Resources. Measured 15:47 today: pid 1294627, 0.5% CPU, 104 MB RSS (coordinator: ~0% CPU, ~102 MB). The ExecStartPre graph probe can hold the start for up to 215 s.

Architecture. The consumer module lives in knowledge_graph (knowledge_graph/conversation_consumer.py); the script is a "tools-owned script" and "D-K7 importer" in packages/integration/design/boundary-census.md (row for deploy/systemd/orchestrata-conversation-consumer.service:33). Component knowledge_graph (+ tools).

7. orchestrata-history-mining.{service,timer}

What it runs. Timer: OnCalendar=quarterly, Persistent=true. Service (oneshot): WorkingDirectory=/opt/orchestrata, .env, PYTHONPATH=/opt/orchestrata/src, ExecStart=/opt/orchestrata/.venv/bin/python scripts/mine_prompt_history.py. Not graph-gated. Code (toolkit --residue, scripts/mine_prompt_history.py): "Quarterly prompt-history drift miner ... mines conversation history for ruling-shaped prompts, clusters them, and diffs the clusters against the ratified rulings canon (rulings/index.md) ... Read-only. ... Its only output is a markdown report, reports/ruling-drift-YYYY-MM-DD.md." Inputs: the KG's SPEC-021 conversation capture (primary), then ~/.kimi/user-history/*.jsonl and ~/.claude/history.jsonl.

Why it was built. plans/workflow-improvements-plan.md Phase 5, Step 5.1: "Problem: corrections repeat across sessions; the export showed the pattern. Catch new ones early." Script docstring: "the same rulings get re-issued verbally for months before they're mechanically enforced." Commit 4116a55b8, 2026-07-26 ("Workflow plan Phases 3 + 5 ...": "Quarterly systemd user timer (orchestrata-history-mining, next fire 2026-10-01) plus a session prime staleness line that fires when the report goes missing - the timer breaking is itself detected."). The unit files were put under version control in 83a3d5fd2, 2026-07-26.

What uses it. session overview "STALE TRACKING" warns "when the newest report is >100 days old" (AGENTS.md, rule 6, "Ruling-drift mining cadence (workflow-improvements-plan.md Phase 5.2)"). A human reads the report to file new rulings ("a human judgment call", script docstring). docs/supervisor-workflow.md §401 refers to it as "the ruling-drift miner pattern".

What breaks while it is off. Nothing at runtime. After 100 days without a report the session overview warns; the only report in the repo is reports/ruling-drift-2026-07-26.md, so the warning would fire from about 2026-11-03. The timer is still enabled and not touched on 2026-10-22; next fire 2027-01-01. Not established: the timer shows a last trigger of 2026-10-05 16:24 but no new report exists and the journal has no entries for the service, so whether that run succeeded is unknown.

Resources. None recorded. Commit d532c38b9 (2026-09-23): it was "the one prod service still bound to /usr/bin/python3 (measured 2026-09-23)".

Architecture. Not in the architecture docs.

8. orchestrata-kg-retention.{service,timer}

What it runs. Timer OnCalendar=Sun 04:00, Persistent=true. Service (oneshot): .env + env/prod.env, ExecStartPre graph probe, ExecStart=.../python scripts/kg_retention.py --retention-days 30 --cxp-exhaust --apply. Code (toolkit --residue): "Recurring KG retention: age out gateway audit exhaust + CXP exhaust." Removes (1) GatewayAuditEvent nodes older than the window and the UUID connection Session nodes they isolate, (2) CXPPacket router test storms and fixture packets, all with work_status pending. "Dry-run by default; --apply deletes in batches and verifies a second pass finds nothing (exit 1 on residue). Idempotent ... Operational KG database only."

Why it was built. The weekly policy predates the repo's unit copy. Rationale recorded: docs/gateway-audit-rollup-2026-07-26.md: the purge was "user-ratified" (session 254); the audit exhaust was "325,366 GatewayAuditEvent nodes + ~250K UUID-keyed connection Session nodes - ~72% of the operational KG at the time, containing no work products or decisions." The script docstring: "both user-ratified 2026-07-26". Commit ea56e6057, 2026-08-02, "Resurrect recurring KG retention as scripts/kg_retention.py": the policy was deleted by commit e73df607 with a one-shot, and "the unit has failed exit 2 on every activation since (surfaced by the boot-health hook 2026-08-02)". Unit versioned in 83a3d5fd2. The original creation of the timer: no recorded rationale beyond the above (the earlier unit lived only in ~/.config/systemd/user, per 83a3d5fd2).

What uses it. The graph itself (size and query cost of orchkg). No code consumer reads its output. The migration plan (Step 1.3) says the timer "keeps working unchanged at cutover".

What breaks while it is off. Nothing immediately. Exhaust accumulates in orchkg; the gateway and test traffic that produced the 72% bloat is the source, so growth depends on the gateway being up. Last run 2026-10-18 04:01: all counts 0, "verified: 0 residue", 83 MB peak, 4.2 s CPU (journal). Timer still enabled, next 2026-10-25 04:00.

Resources. 4.2 s CPU, 83.1 MB peak per run (journal 2026-10-18). Unit bug history: ca8a65965 2026-10-07 (it gated a prod sweep on the test graph: "every timer run waited 120s on bolt://10.53.2.10:7687 ... and failed while the prod cluster was healthy").

Architecture. Not named in the architecture docs; it operates on the operational KG that component-dag.md calls "Prod orchkg (database orchkg)".

9. orchgraph-backup.{service,timer}

What it runs. Timer OnCalendar=*-*-* 03:00:00, Persistent=true. Service (oneshot): .env + prod profile, ORCHGRAPH_BACKUP_DIR=/var/lib/orchestrata/backups/orchgraph, ExecStartPre graph probe, ExecStart=/opt/orchestrata/scripts/backup_orchgraph.sh --retain 14, output appended to backup.log. Script header (shell, read freely): resolves the current MAIN instance, runs CREATE SNAPSHOT; on every database ("the default memgraph database (semantic/RAG graph) AND orchkg (the operational KG)"), incus file pulls each newest snapshot, prunes to the newest N runs. "Memgraph takes a consistent snapshot online ... so this script never interrupts service."

Why it was built. Migration plan Step 1.3 "Backup + retention plumbing" (plans/neo4j-to-orchgraph-migration-plan.md): replace the Neo4j backup, "daily 03:00, same slot - trigger CREATE SNAPSHOT;, incus file pull the snapshot dir, prune to N=14". Commit cb26d0167, 2026-07-26, "Migration Step 1.3: orchgraph snapshot backup + daily timer, restore-drilled" ("a scratch container built from the orchkg snapshot file alone came up with 71 constraints + 76 indexes, matching the source exactly"). Wiki deploy/systemd/README.md: "Since the cutover this is the ONLY backup covering the operational KG." ADR-0072 cites it as "the nightly orchgraph-backup.service snapshots every database on that cluster".

What uses it. Nothing reads the archive in normal operation (restore drill only). It is the safety net for every writer to orchkg (gateway, kernel, ast-live-sync, conversation-consumer).

What breaks while it is off. No new snapshots; the 14 retained runs stop rolling forward. Unit is not disabled: it ran today 03:00:38, success; manifest {"databases":{"memgraph":19973,"orchkg":156953},"main":"10.53.1.123:7687"}. A notable history: commit 8b2038b92, 2026-10-05, "orchgraph backup has never produced a snapshot - three defects": "The orchgraph-backup unit failed on every run since it was installed; the archive ... held only backup.log." So between 2026-07-26 and 2026-10-05 there was effectively no backup (as stated in that commit; the archive directory dates start 2026-10-09 because retention keeps 14 runs).

Resources. Payload size, from plans/system-analysis-2026-08-23.md: "backups/ | 8.9 GB untracked | the payload orchgraph-backup.service writes into the dev tree" (before the archive moved to /var/lib/orchestrata/backups/orchgraph). Each pull is large (e.g. 221.81 MB for memgraph on 2026-10-22 per the log); the root NVMe vs T9 budgets are separate (user memory). The 03:00 run touches the T9 pool.

Architecture. Not named in the architecture docs (data layer of the orchkg database).

10. orchestrata-infra-drift.{service,timer}

What it runs. Timer OnCalendar=hourly, RandomizedDelaySec=120, Persistent=true. Service (oneshot): no .env, no kg-ready; ExecStart=/opt/orchestrata/.venv/bin/python /opt/orchestrata/scripts/check_infra_drift.py --report /var/lib/orchestrata/terraform/drift-latest.json. Code (toolkit --residue): "Infra drift gate - the mechanism for ADR-0077 item 4." Runs tofu plan -detailed-exitcode over infra/terraform (exit 0 zero-diff, 2 drift), and proves three bridge keys the provider never stores via incus network get. "A missing state is a failure, never a skip: the September state was lost to a worktree gc and nothing noticed." Exit 1 on drift.

Why it was built. ADR-0077 (Accepted, Mohamed, 2026-10-08: "all environments must be imported"), item 4 "The drift gate fails loudly", and the stated gap: "The drift gate is silently off. scripts/health_check.py check_infra_drift returns PASS with a note whenever terraform.tfstate is missing ... and no timer runs the health check." Timer description: "Hourly Orchestrata infra drift gate (ADR-0077 item 4)"; ADR text: "check runs on a user timer so drift is detected, not discovered." Added in commit f2cef12cb, 2026-10-08 ("feat(infra): execute ADR-0077 ... drift gate + timer, boot.autostart applied").

What uses it. scripts/health_check.py shells to the same script (commit f2cef12cb). The JSON verdict is "kept beside the state for the error report and the next run to read" (unit comment). Gate named in ADR-0110: "check_infra_drift ... green at zero diff afterwards".

What breaks while it is off. The Incus fleets (prod default, orchestrata-test, orchestrata-dev) are no longer checked hourly against OpenTofu state; drift is again "discovered, not detected". Current verdict, which is why it matters today: the last run (2026-10-22 15:01:27 EDT, drift-latest.json) reported verdict: drift, plan_exit: 2, "Plan: 3 to add, 14 to change, 0 to destroy", the 3 additions being incus_instance.prolog["node-1".."node-3"], with bridge_drift: []. The service exited 1/FAILURE at 14:31 and 15:01. Journal pattern: 23-24 successful runs per day on 2026-10-14..19, one failure on 10-20, two on 10-22. The cause was measured (coordinator, tofu plan -refresh-only, read-only, 2026-10-22 ~14:45). All 17 items are changes made outside OpenTofu:

• boot.autostart "true" -> "false" on 13 instances: the 3 dev, the 9 test and test_app.
• restricted.devices.disk.paths on incus_project.test: the old checkout path dropped, /opt/orchgraph-nvme added (the NVMe move of 2026-10-20, reverted 2026-10-21; that path is now unused).
• prolog-node-1..3 deleted.

Mohamed confirmed on 2026-10-22 that he set autostart off and deleted the containers ("I deleted unneccessary containers my machine is cpu maxed and the t9 ssd with btrfs was thrashing"). The prolog deletion is also backed by ADR-0110. Reconciling infra/terraform with this reality is pending.

Resources. 1.2-4.3 s CPU and 82-146 MB peak per run (journal). Mohamed stopped the timer to cut host CPU (coordinator).

Architecture. Not in component-dag.md or architecture-ontology.md. Its governing doc is ADR-0077 and infra/README.md (infrastructure layer, outside the ten components).

How they are deployed

• Sources live in deploy/systemd/*.service|*.timer in the repo (version-controlled since 83a3d5fd2, 2026-07-26: "deploy/systemd/ now holds all seven units, byte-identical to installed"). Every prod unit executes from /opt/orchestrata (ADR-0062; RUL-027). Template units (orchestrata-ast-daemon@, orchestrata-codemod-daemon@) and orchestrata-canvas.service are not part of the ten and are not enabled by the installer.
• deploy/systemd/install.sh copies changed unit files to ~/.config/systemd/user, reloads systemd, and is run by a promote's assets stage (src/orchestrata_ultimate/promotion/stages_assets.py, which looks up ctx.target_root/deploy/systemd/install.sh; the runner also names it, promotion/runner.py:178).
• Until a pending fix (uncommitted edits to deploy/systemd/install.sh, deploy/install-orchestrata.sh, deploy/ship.sh, promotion/stages_units.py in the worktree), the committed install.sh also ran systemctl enable --now on every timer and every long-running service (install.sh line 144; header: "Enables + starts every timer, and every service that is a long-running daemon"). That is why the six services and the timers were enabled at all, and why a promote re-enables what Mohamed disables. The 2026-10-22 re-install used --bootstrap-only and touched no unit (coordinator).
• Readiness: deploy/orchestrata-kg-ready.py as ExecStartPre on seven units (bot, ast-live-sync, conversation-consumer, gateway, kernel, kg-retention, orchgraph-backup), 215 s budget since 0d772955a (2026-10-13).
• Drift: scripts/health_check.py check_systemd_units fails if an installed unit differs from the repo copy, or a declared unit is not installed (commit 83a3d5fd2).
• Not units in this set: the host hook irc_agents_hook.sh starts miniircd.service but never enables it.
• Tool units: install-tools.sh installs the orchestrata-ast-daemon@ and orchestrata-codemod-daemon@ templates and runs deploy/retire-plain-units.sh, which removes the old checkout-bound plain units. After a promote, deploy/restart-tool-instances.sh restarts every active instance. It never enables a new instance.
• Conflict with ADR-0123. ship.sh step 5b fails the ship unless every /opt/orchestrata unit is live, and it fails on 0 live units. ADR-0123 says a deploy never enables or starts a unit. While the units DeepChat disabled stay disabled, a full ship cannot pass step 5b. See ship and deploy.

Decisions on 2026-10-22

All from the coordinator; no further evidence gathered.

• 15:31: Mohamed had all six services (miniircd, bot, kernel, gateway, ast-live-sync, conversation-consumer) and orchestrata-infra-drift.timer stopped and disabled, to cut CPU load (the machine was CPU-bound and the T9 btrfs pool was thrashing). Reasons given, not tied to a single unit.
• On 2026-10-20 he had turned off fleet boot.autostart and deleted unneeded containers. He confirmed this on 2026-10-22: "yes, I did it; I deleted unneccessary containers my machine is cpu maxed and the t9 ssd with btrfs was thrashing".
• ADR-0123 (Proposed) proposes this page as the register of host services, with a keep, on-demand or retire disposition per unit, ruled one unit at a time.
• 15:35: prod /opt/orchestrata re-installed clean at 4ac0db570 with install-orchestrata.sh --bootstrap-only; no unit was touched.
• 15:39: orchestrata-conversation-consumer was started (not enabled) at his request. It flushed 51 queued turns into prod orchkg, then idles at about 0% CPU and ~102 MB RSS.
• The SessionStart IRC hook starts miniircd.service whenever it is down; it starts it and does not enable it.
• The orchestrata MCP server answers gateway_ping with "GATEWAY_UNAVAILABLE: GATEWAY_TOKEN is not set".
• Still enabled and running on their schedule: orchgraph-backup.timer, orchestrata-kg-retention.timer, orchestrata-history-mining.timer.

Open questions

1. Why IRC (miniircd + bots) was chosen as the fleet bus, and why orchestrata-bot exists: no recorded rationale found.
2. Whether anything else depends on #agents or #orchestrata besides the Claude listener and the IRC conversation producer: not established. A listener (pid 11265) is currently retry-looping against the closed port.
3. What the gateway does in its PipelineKernelClient fallback when the kernel is off (which RPCs lose function): not traced.
4. Which hooks and tools besides orchestrata-mcp and VizInt call the gateway on this host: not enumerated (ADR-0065 says "the CLI, hooks, the kernel").
5. Whether the prod kernel entry agents.runtime_main also boots the conversation consumer (only kernel/cli_main.py imports it directly): not traced; matters for whether the consumer unit is the sole drainer.
6. Where REQ-021-069/070 and US-021-007 (conversation capture) are specified: specs/021-opencode-integration/spec.md is the OpenCode spec; the capture requirements are cited by code and ADR-0072 but their text was not located.
7. History-mining: the 2026-10-05 16:24 trigger left no report and no journal lines; whether it failed or the journal rotated is unknown. The 100-day staleness warning would fire around 2026-11-03.
8. Infra-drift: answered. The 14 in-place changes are 13 boot.autostart true->false plus the test project's disk paths (see section 10). What is still open is reconciling infra/terraform with that reality and with ADR-0110. With the timer disabled, the last report stays at DRIFT.
9. ADR-0062 shows "Status: Proposed (awaiting Mohamed's ruling)" in its file header while RUL-027 and deploy/systemd/README.md treat it as in force; its ratification record was not located.
10. Backup gap: 2026-07-26 to 2026-10-05 had no successful snapshot (commit 8b2038b92); whether a copy from that period exists elsewhere is unknown.
11. Added 2026-10-23: two GitLab runners are active, the user unit and the system unit, with different config files (~/.gitlab-runner/config.toml and /etc/gitlab-runner/config.toml). Which of them is meant to serve CI has not been ruled.
12. Added 2026-10-23: the user units outside orchestrata (legacy MCP servers, callback-agents, knowledge-sync, local-mcp-server, mcp-server, nightly-test-runner, audit-service) have no architecture home. Under ADR-0123 they are retirement candidates. None has been ruled.
13. Added 2026-10-23: performance-monitor.timer is enabled and fires every 15 minutes. It runs ~/enterprise-python/config/system_monitor.py. Its purpose and its CPU cost were not measured.
14. Added 2026-10-23: the codetools verify suite left a transient fixture unit, codetools-_TEST_cc4c05dd_78e4153173c9.service, in /run/user/1001/systemd/user/. It disappears at logout. Whether the suite should clean it up is a codetools question; see codetools blockers.

Related

• Deployment: every deployment, with its installer, verification and rollback
• Deployment inventory
• ship and deploy
• Incus fleets
• Gateway, Kernel, Redis
• Memgraph HA cluster reference
• Open decisions
• Decisions
• Claude hooks
• codetools MCP