Home / Projects / Orchestrata Ultimate / Decisions / ADR index / ADR-0067

ADR-0067: Developer daemons are deployed code; the checkout they serve is a runtime parameter

← ADR-0066 · ADR-0068 →

The record

Status: Accepted (rulings R1/R2 of plans/ast-daemon-deployment-plan.md, Mohamed, 2026-10-05: "1. every deploy must follow the professioal deployment menthods and be seperate from code, 2. yes codemodd should be fiuxed too") · Date: 2026-10-05

Context

Two long-running daemons accelerate development on this host: the AST daemon (scripts/ast_daemon.py, warm symbols/refs/source over a Unix socket) and the codemod daemon (orchestrata_ultimate.refactor.codemod._daemon_entry, the SPEC-110 code-intelligence-graph query server). Measured 2026-10-05:

• Both run as plain user units in ~/.config/systemd/user (orchestrata-ast-daemon.service, orchestrata-codemod-daemon.service) whose ExecStart and WorkingDirectory name the ~/workspace/orchestrata-ultimate-architecture checkout and its .venv. That checkout is on branch architecture/domain-model at d4b394279 (2026-09-23), dirty, and is not the primary checkout.
• Neither unit is in the repository. The repo ships only TEMPLATE units (deploy/systemd/orchestrata-{ast,codemod}-daemon@.service) whose ExecStart is /%I/.venv/bin/python … — the served checkout's own venv — started runtime-only by scripts/dev_daemons.sh. The "runtime-only, never ~/.config" rule (labelled U-T2) exists only as a comment in the AST unit's header; ADR-0062's text does not contain it.
• Nothing records which commit either daemon runs, nothing installs them from a pinned ref, and nothing refuses success when the running process is not the deployed one. Both survived the 2026-10-05 prod promote/rollback untouched; the AST daemon went inactive during it and nobody was told.
• Each Claude session's start-ast-daemon.sh hook additionally spawns an AST daemon from the session checkout's venv, detached with setsid.
• Footprint: AST daemon 546 MB RSS at 4 min (1.7 GB recorded 2026-08-30 in the unit header); codemod daemon 1.9 GB RSS at 5 h 40 min.
• /opt/orchestrata-tools (deploy/install-tools.sh) already deploys the project-agnostic tools (ast_live_sync, observe_pytest, swarm_supervisor, ship) as copies under lib/ with their own python3.14t venv, a VERSION file recording the source commit, bin/ entry shims, and a verifier that refuses workspace paths. Its venv holds neo4j and watchfiles; the daemons also need fastapi uvicorn pydantic (AST) and psutil xxhash codemod_fast (codemod).

This is the platform's deploy-in-place defect (CLAUDE.md convention 4, RUL-027) in three forms at once: code executing from a working tree, no identity, and configuration carrying a checkout path.

Decision (Mohamed, rulings R1/R2)

1. A daemon that serves developer checkouts is deployed code. The AST daemon, the codemod daemon, and any future daemon of this kind are installed under /opt/orchestrata-tools by deploy/install-tools.sh from a pinned ref, into that deployment's own venv, at the commit VERSION records. The checkout a daemon serves is a runtime parameter — the systemd instance name %I (an escaped path) and the daemon's --repo-root / repo_root argument — never the location of the code it executes.

2. No daemon process on this host executes from a workspace checkout. Not a plain unit, not a template instance whose ExecStart points at /%I/.venv, not a hook-spawned process. When the tools deployment is absent, the SessionStart hook says so in one line and the AST client takes its existing slow path; it does not spawn a daemon from the checkout venv. The U-T2 "runtime-only" compromise existed because a checkout literal in config was the code location; with code under /opt, an enabled instance naming the served checkout is data, and is permitted for the primary checkout (~/workspace/orchestrata-ultimate). Worktree instances stay runtime-only, started by the session hook.

3. Identity, not health. The AST daemon's /health and the codemod daemon's JSON-RPC status report code_root and deployed_commit. The installer's verify step starts each deployed daemon against a fixture checkout and asserts code_root is under the deployment and deployed_commit equals the shipped SHA; the session hook REPLACES a running daemon whose code_root is a working tree.

4. The gate refuses their return. scripts/check_prod_tree_separation.py (RUL-027) also inspects ~/.config/systemd/user/*.service: a non-template unit whose ExecStart or WorkingDirectory resolves under /home/*/workspace/ is a violation. install-tools.sh refuses success while orchestrata-ast-daemon.service or orchestrata-codemod-daemon.service (plain) exists.

5. The codemod daemon is in the same wave (R2), as the second half of each packet, after the AST daemon proves the shape.

Alternatives rejected

• Keep the plain units, add a drop-in pointing at the primary checkout. Still deploy-in-place; still no recorded commit; still nothing that refuses a wrong process. It would have made the symptom go away and the defect permanent.
• Deploy the daemons into /opt/orchestrata (the prod tree). They serve developer checkouts and must not depend on prod promotion, which is gated on the prod fleet and on open operational decisions (ADR-0066 R16b). The prod tree is the product; /opt/orchestrata-tools is the harness, and it already owns the sibling tools.
• Keep the hook's checkout-venv fallback for hosts without the deployment. Rejected by R1: a fallback that runs code from the working tree is the pattern being removed. The client's slow path already keeps correctness.
• Runtime-only instances everywhere (no enable). Nothing warm after a reboot until a session opens; the old plain unit's one virtue lost. The reason for runtime-only no longer applies once the instance name is data.

Consequences

• install-tools.sh grows: venv deps fastapi uvicorn pydantic psutil xxhash (measured against a clean venv), the codemod_fast native module built with the pinned system toolchain, bin/ast-daemon and bin/codemod-daemon shims, installation of both template units into ~/.config/systemd/user, and a live smoke test per daemon.
• deploy/systemd/orchestrata-{ast,codemod}-daemon@.service: ExecStart begins with /opt/orchestrata-tools/bin/, WorkingDirectory is the deployment, ExecStartPre asserts /%I/.git, MemoryMax is set from measurement (provisional 3G AST, 4G codemod). scripts/dev_daemons.sh keeps working unchanged.
• scripts/ast_daemon.py gains --repo-root and identity in /health; _daemon_entry already takes repo_root and gains identity in status.
• .claude/hooks/start-ast-daemon.sh (and its plugin mirror) becomes a two-branch hook: healthy deployed daemon → done; otherwise systemctl --user --runtime start orchestrata-ast-daemon@<escaped> (after stopping a workspace-bound one); no deployment → one line, slow path.
• The two plain units are disabled and removed as part of the P5 ship's verify, not by hand.
• Memory: N instances × (≤1.7 GB AST + ≤1.9 GB codemod). Mitigations: MemoryMax, idle exit (AST), instances only for checkouts a session opens; fs.inotify.max_user_watches measured against instance count before P6.
• The plugin reference copies under plugins/orchestrata/systemd/ are regenerated by gen_plugin, never edited by hand.

Provenance

• Plan and packets: plans/ast-daemon-deployment-plan.md (committed d6c8e487a, amended with these rulings in the following commit).
• Rulings R1/R2: Mohamed, 2026-10-05, in the session that promoted prod and found the AST daemon inactive afterwards.
• Measurements: systemctl --user show -p MainPID, /proc/<pid>/cmdline, /proc/<pid>/cwd, /proc/<pid>/status on 2026-10-05; git -C ~/workspace/orchestrata-ultimate-architecture log -1 / status -sb; /opt/orchestrata-tools/.venv/bin/python -c 'importlib.util.find_spec' for the venv inventory.
• Related: ADR-0062 (prod is a pinned checkout under /opt/orchestrata), RUL-027 (prod/dev tree separation), ADR-0066 (secrets file; R16b).

Related ADRs

• Also mentioned: ADR-0062, ADR-0066

Rulings cited

• RUL-027: RUL-027 — Production runs its own checkout, never the development tree

Specs cited

• SPEC-110