Orchestrata Ultimate › Plans › ast-daemon-deployment-plan

PLAN — the AST daemon as a real deployment

Source: plans/ast-daemon-deployment-plan.md at 8a2cb42ad · last changed 2026-10-07 · first committed 2026-10-05 · 1257 lines.

Prose pending.

Plan text

PLAN — the AST daemon as a real deployment

Status: RULED, ready to build (2026-10-05). D1 and D2 below were decided by Mohamed the same day (rulings R1/R2, recorded as ADR-0067); every packet is now engineering under the platform conventions. R2 puts the codemod daemon in the same wave, so each packet has an AST half and a codemod half.

0. Measured state (2026-10-05, this host)

Three distinct violations of the platform conventions:

1. Deploy-in-place. The running daemon executes a working tree's code and venv (ExecStart=…/workspace/…/.venv/bin/python scripts/ast_daemon.py).
2. No identity. Nothing records which commit it runs, no installer resets it to a pinned ref, and no verifier refuses success when the running thing is not the deployed thing. It went inactive and nobody was told.
3. Config carries a checkout path. The unit is persistent, version-less wiring that names a developer checkout — the exact thing U-T2 removed from the repo, surviving on this host outside the repo.

1. Target

The AST daemon is code deployed once, serving N checkouts:

• Code lives under /opt/orchestrata-tools (already the home of every project-agnostic tool), in its own uv venv on python3.14t, at the commit VERSION records, installed by deploy/install-tools.sh from a pinned ref.
• Data is the checkout being served. It is a parameter of the unit instance (%I), never a path inside the code or the unit file.
• Unit orchestrata-ast-daemon@.service is version-controlled in deploy/systemd/, installed into ~/.config/systemd/user by the installer, and executes only /opt/orchestrata-tools/....
• Identity, not health: /health reports the code's deployed commit and the served repo root; the installer's verify step and the SessionStart hook refuse/replace a daemon whose code root is not /opt/orchestrata-tools.
• The checkout-bound plain units (orchestrata-ast-daemon.service, orchestrata-codemod-daemon.service) are gone from this host and a gate refuses their return.

2. Decisions — RULED (Mohamed, 2026-10-05, ADR-0067)

D1 — may an enabled instance persist the served checkout path? Asked because systemctl --user enable orchestrata-ast-daemon@<escaped-path> writes a symlink naming the checkout, and U-T2 had made dev instances runtime-only on the ground that a checkout path in config was deploy-in-place.

R1 (Mohamed): "every deploy must follow the professional deployment methods and be separate from code." Applied: the daemon's code lives only under /opt/orchestrata-tools; the served checkout is a runtime parameter (%I, --repo-root), i.e. data. An enabled instance therefore names data, not code, and option (a) stands: enable for the primary checkout only; worktrees stay runtime-only via the hook. R1 also removes the checkout-venv fallback I had kept in P4 — a process executing from a working tree is the defect, whatever starts it; without the deployment the client takes its slow path. Recorded as ADR-0067 (ADR-0062 is still Proposed and never carried the runtime-only clause, so a new record, not an amendment).

D2 — codemod daemon in the same wave? R2 (Mohamed): yes. orchestrata-codemod-daemon.service has the identical defect (plain unit, checkout-bound; measured: -m orchestrata_ultimate.refactor.codemod._daemon_entry <repo> <repo>/.codemod_daemon/socket <repo>/.codemod_daemon/pid from the architecture checkout's venv, cwd there, RSS 1.9 GB at 5 h 40 min, enabled). Each packet below carries a codemod half; the AST half lands first within the packet so the shape is proven before it is copied.

Codemod facts the halves rely on (measured 2026-10-05): the entry already takes repo_root socket pid_file positionally, so it is location-independent except for its imports; it imports codemod_fast (native, built from scripts/_codemod_fast/, with a pure-Python fallback the tests cover), psutil and xxhash, none of which the tools venv has; the client finds the socket at <repo>/.codemod_daemon/socket; CodemodDaemon.start(repo_root) spawns the entry with sys.executable; no session hook starts it today; scripts/dev_daemons.sh manages both templates together.

3. Packets

Each packet: one worktree off origin/master, one brief, gates green, ship. Opus decides, Sonnet builds. Numbers in acceptance criteria are to be MEASURED and written into the commit message. Each packet has an AST half (listed first) and a codemod half (↳ codemod:), shipped together unless the codemod half is blocked, in which case the AST half ships and the blocker is reported — never silently dropped.

Packet Definition of Done (ADR-0069 / RUL-034, Mohamed 2026-10-06, applies to P4–P7 from this date; P1–P3 shipped before the ruling and owe the gap in P7). Five items in the same ship as the code: (1) @traces on every added/changed symbol and test, spec-defined ids only; (2) RTM regenerated (generate_rtm_evidence.py, trace_intel.py untraced-reqs before/after); (3) checklist/docs//design-status updated in the code commit; (4) the knowledge-wiki page for the daemons updated and shipped; (5) ≥90% line coverage per touched source file, measured. The ship log gains the five as columns from P4 on.

P1 — make the daemon location-independent (code)

scripts/ast_daemon.py, scripts/ast_api.py

• Add --repo-root PATH (required when cwd is not a repo; default: cwd). All source/cache/socket defaults resolve against it; the daemon chdirs nowhere and never reads Path.cwd() after start-up.
• /health gains repo_root, code_root (= Path(__file__).resolve().parents[1]), deployed_commit (read from <code_root>/VERSION when present, else null), pid.
• Socket default: <repo_root>/.claude/run/ast-daemon.sock (unchanged for callers).
• Finding during build (2026-10-05, builder, no edits made): the cwd dependence is not in ast_api.py; it is in scripts/serialize_codebase_ast.py. _iter_py_files(sources) resolves a relative source against the process cwd and every _QUERY_DISPATCH handler goes through it; the client sends relative source strings; result file fields are repo-relative only because the CLI runs inside the checkout. Separately, _TOOLKIT_REPO_ROOT = Path(__file__).resolve().parent.parent (the CODE root) is used by refs alias resolution (_owning_module_for), so a deployed toolkit serving another checkout would map definitions against the wrong tree. Three ways out were weighed: (1) rewrite paths in daemon answers back to repo-relative — a recursive string rewrite that can touch source snippets; (2) a seam in the toolkit; (3) absolute paths in daemon answers — a wire-output change. Engineering pick (not a ruling: no observable contract changes; it executes ADR-0067 §1 inside the toolkit): option 2. The toolkit gains one explicit data-root seam — configure_repo_root(path) (module-level, idempotent, default = today's behaviour so the CLI is unchanged) — and _iter_py_files, the refs owning-module resolution and any other _TOOLKIT_REPO_ROOT-as-data use resolve relative sources against it and emit paths relative to it. The daemon calls it once with repo_root before serving. _TOOLKIT_REPO_ROOT keeps its one legitimate meaning (where the toolkit's own code lives) for code-root uses only. P1's FILES therefore include scripts/serialize_codebase_ast.py, and the byte-identical property is asserted by test 3(b) running the daemon from a foreign cwd.
• Review of the first P1 commit (2026-10-05, reviewer, read-only): seam sound (92/92 API parity probes identical across 23 functions × 4 source shapes; 10 CLI invocations byte-identical to master with no root configured). Identity reporting had defects, fixed in the follow-up commit: (F1) install-tools.sh stamps $PREFIX/VERSION while the daemons' code root is $PREFIX/lib, so deployed_commit was always null in the real layout — rule: deployed_commit is read from <code_root>/VERSION, then <code_root>/../VERSION; the first that exists wins. Both halves follow it. (F2) VERSION was re-read on every /health, so a redeploy made a stale process report the new SHA — rule: identity is captured once at process start and never re-read. (F3) a nonexistent --repo-root served an empty tree with a healthy /health and created directories under the typo — the daemon refuses to start unless repo_root exists and is a directory. (F4) the parser default resolved Path.cwd() eagerly and crashed when the cwd had been deleted even with --repo-root given. (F5) no test used a relative --socket/--cache or exercised the watcher from a foreign cwd. (F6) GET /check-deps in ast_api.py resolved pyproject.toml against cwd (500 from a foreign cwd). (F7) commit regex accepted any word and rejected trailing whitespace → [0-9a-f]{7,40} on stripped lines. (F8) one parity loop could pass on empty results → non-empty asserted. (F9) the GET routes mounted from ast_api ran the toolkit without cache_lock while the watcher mutates the parse cache (predates P1; fixed in the same P1 follow-up commit under RUL-024, every toolkit-reaching request now takes the lock /query already takes, /health stays outside it, proven by a held-lock test). Reviewer re-check of the fix commit (2026-10-06): F1–F9 CONFIRMED-FIXED by the original probes; CLI parity identical; API parity 92/92. Three nits closed in a final P1 commit: deleted cwd without --repo-root exits 2 with a message instead of a traceback; /healthz stays outside the route lock like /health; the F5 test asserts the path _prepare actually configures, not the helper; and a VERSION file without a commit line falls through to the next location instead of hiding it. Engineering item carried to P6 (reversible, pre-dates P1, no contract): the daemon never writes the parse-cache sidecar (_save_sidecar is CLI-only), so --cache only governs a warm-start read of a file nothing in the daemon produces. P6 measures warm-start time with and without a sidecar before anything changes. Moved to P4 as scope, not as a defect: the client (query_ast_daemon) accepts any daemon that answers /health; P4 makes it compare repo_root and code_root because that is the packet that changes the client.
• Tests (tests/scripts/test_ast_daemon.py): start the real daemon from a foreign cwd against a fixture repo under tmp_path; /health names the fixture as repo_root and the script's parent as code_root; a query answers the same as serialize_codebase_ast.py run locally on the fixture.
• Acceptance: the affected-tests blast radius for scripts/ast_daemon.py and scripts/ast_api.py (as scripts/affected_tests.py computes it) green; new tests green; no reference to Path.cwd() left in the daemon after argparse.
• ↳ codemod: orchestrata_ultimate/refactor/codemod/{_daemon_entry,server,daemon}.py. The entry is already parameterised; add an identity JSON-RPC method (or fields on the existing status reply — pick whichever the client already polls, do not add a second poll) returning repo_root, code_root (= the orchestrata_ultimate package's parent), deployed_commit (<code_root>/VERSION or null), pid. CodemodDaemon.start(repo_root) keeps spawning with sys.executable for now (P4 changes who starts it). Tests in tests/refactor/codemod/test_daemon_entry_coverage.py: real daemon from a foreign cwd against the fixture package; identity names the fixture and the package root. Acceptance: codemod blast radius green.

P2 — deploy the daemon with the tools (installer)

deploy/install-tools.sh

• Venv deps: add fastapi uvicorn pydantic beside neo4j watchfiles (measured against a clean venv, as the existing comment demands).

• write_entry ast_daemon ast_daemon.py → /opt/orchestrata-tools/bin/ast-daemon (the same exec shim shape as ast_live_sync). Measured after the P1 ship (2026-10-06): the premise that lib/scripts/ already holds the toolkit was false — the installer copies a curated six-script list (swarm_supervisor, observe_pytest, ast_live_sync, build_runtime_callgraph, orchestrata_pytest_observer, __init__), so P2 adds scripts/ast_daemon.py, scripts/ast_api.py, scripts/serialize_codebase_ast.py and whatever those import from scripts/ (enumerate with who-imports/refs, do not guess) to that list, and scripts/_codemod_fast/ for the codemod half. Confirm by --help from the deployed path.

• Verify step, in order, each one fail on miss:

  1. bin/ast-daemon --help exits 0;
  2. the entry resolves under $PREFIX, never a workspace path (existing checks);
  3. live smoke: start bin/ast-daemon --repo-root <tmp fixture> --socket <tmp> detached, poll /health ≤ 30 s, assert code_root == $PREFIX/lib, deployed_commit == $SHA, run one /query, SIGTERM, assert clean exit;
  4. grep -q "^commit $SHA$" VERSION stays last (D1 of install-tools: staged then promoted after every check).

• Tests (tests/deploy/test_install_tools.py or new test_install_tools_ast_daemon.py): run the installer's render + verify arms against a tmp_path prefix (the installer needs a PREFIX override seam if it has none — add one, env-read via ${PREFIX:-/opt/orchestrata-tools}).

• Acceptance: /opt/orchestrata-tools/VERSION at the shipped SHA; bin/ast-daemon answers /health with that SHA from a cwd outside any repo.

• ↳ codemod: venv deps psutil xxhash; build codemod_fast from lib/scripts/_codemod_fast/ into the tools venv with the pinned system toolchain (worktree_toolchain.pin_system_toolchain, the same seam the build scripts use since 078fb669f); bin/codemod-daemon shim running python -m orchestrata_ultimate.refactor.codemod._daemon_entry "$@" from the deployment's lib/. Verify: --help/import check; live smoke against a fixture package asserting identity (code_root == $PREFIX/lib, deployed_commit == $SHA) AND that the native codemod_fast loaded — the pure-Python fallback loading in the deployment is a verify FAILURE, not a degraded pass. Acceptance: both daemons answer identity with the shipped SHA from a cwd outside any repo.

• P2 build record (2026-10-06, commits ec00c8b5c + 8982850ab). Premises found false and how they were resolved: (1) orchestrata_ultimate under lib/ carried only core and knowledge_graph — the archive list now includes src/orchestrata_ultimate/refactor (codemod + ctok). (2) ctok is a second native dependency (refactor/ctok/tokenizer.py imports it) — built from scripts/_ctok the same way as codemod_fast (uv pip install --no-build-isolation, CC=/usr/bin/gcc CXX=/usr/bin/g++, Guix include vars dropped; needs network or a warm uv cache for setuptools wheel). (3) a flat write_entry shim would make code_root equal $PREFIX, not $PREFIX/lib — bin/ast-daemon runs lib/scripts/ast_daemon.py with PYTHONPATH=$PREFIX/lib, and the smoke caught the missing path as ModuleNotFoundError: orchestrata_ast. (4) the daemons read $PREFIX/VERSION, which the installer promotes LAST, so the smoke copies the staged record into place for its duration and a cleanup trap restores the previous record (or removes the file) on any failure; test (d) asserts a refused install leaves no VERSION. Also: PREFIX seam (${PREFIX:-/opt/orchestrata-tools}), $SUDO empty for a user-writable prefix, LINK_DIR=$PREFIX/links for any non-default prefix so a smoke can never repoint /usr/local/bin, codemod_fast_native added to the codemod identity (a False is a verify failure). Only the three toolkit scripts import nothing else from scripts/, so the curated list grows by exactly those plus the two native source dirs. 77 tests / 80 s over five deploy modules; test_install_tools.py has 4 tests at ~25 s each (one real install per test).

• INCIDENT during the P2 build (2026-10-06 02:29). The builder's first draft of test (d) executed the OLD installer out of git archive HEAD; that installer hardcoded /opt/orchestrata-tools and sudo is passwordless, so the real deployment was reinstalled from a throwaway clone whose commit (4065f384d…) exists nowhere, with VERSION recording repo /tmp/p2t-…/clone. The files were the normal tools (same archive list) and /usr/local/bin links still resolved, so nothing broke functionally, but the deployment's identity was false until the P2 ship reinstalled it from the pushed commit. Guard added: the test refuses to execute any installer whose text lacks the PREFIX="${PREFIX:- seam, and test (d) copies the working-tree installer into its clone. Standing rule from this: a test never executes an installer it did not read first, and the brief's "PREFIX=tmp only" is enforced by the test, not by trust.

• First P2 ship (2026-10-06 02:47, master a1dd0e866) — deploy verified, exit non-zero. The ship's verify stage was green (350 passed / 0 failed, 33 modules, 74 s), master fast-forwarded to a1dd0e866, the Claude-config deploy verified, and install-tools.sh reinstalled /opt/orchestrata-tools with every smoke OK (both shims resolve under /opt only; /health reported code_root=/opt/orchestrata-tools/lib deployed_commit=a1dd0e866; the codemod daemon reported the same identity with codemod_fast_native=true) and promoted VERSION to a1dd0e866 — the clobbered record from the build incident is repaired. Then the installer's EXIT trap failed: the native build runs under $SUDO for the real prefix and setuptools wrote root-owned build/ and *.egg-info into the user-owned staging dir, so the trap's plain rm -rf "$STAGE" hit EACCES AFTER DEPLOYED, the script exited 1, ship reported FAIL: install-tools.sh and stopped before the gitlab-runner deploy and the prod promote. Root cause: every P2 test runs with a user-writable PREFIX where $SUDO is empty, so the sudo path was never exercised. Fix: the trap removes the stage with $SUDO rm -rf "$STAGE" and returns 0; pinned by test_cleanup_removes_the_stage_with_the_same_privilege_that_built_in_it (a text pin — the sudo path is unreachable from a tmp-prefix test, stated in the test). The root-owned leftover /tmp/tmp.Ezq7yU3qEq was removed by hand. Lesson for P3–P7: a code path that only the real prefix takes ($SUDO non-empty, /usr/local/bin links) needs its own named verification, not inference from the tmp-prefix tests.

P3 — the unit executes deployed code only (unit)

deploy/systemd/orchestrata-ast-daemon@.service

• ExecStart=/opt/orchestrata-tools/bin/ast-daemon --repo-root /%I --socket /%I/.claude/run/ast-daemon.sock
• WorkingDirectory=/opt/orchestrata-tools (never /%I).
• ExecStartPre=/usr/bin/test -e /%I/.git (-e, not -d: a worktree's .git is a file) (an instance for a vanished checkout fails loudly, it does not spin).
• Keep Restart=always + RestartSec=10 (documented reason in the unit: idle exit is clean). Set --idle-timeout 0 for the enabled primary instance for the enabled primary instance only if measurement shows restart churn; otherwise leave the 2 h default.
• Memory: MemoryMax=3G per instance (1.7 GB peak recorded in the unit header; a runaway must not take the box).
• Installer: install-tools.sh installs this unit into ~/.config/systemd/user (content-compared, daemon-reload on change) — the same mechanism deploy/systemd/install.sh uses for the prod units; reuse, do not duplicate.
• scripts/dev_daemons.sh keeps working unchanged (it installs the same template into the runtime unit dir; after this packet the template no longer references /%I/.venv).
• Tests: tests/deploy/test_deploy_assets.py parses it already; add an assertion module tests/deploy/test_ast_daemon_unit.py: ExecStart begins with /opt/orchestrata-tools/, contains no .venv, no /%I/.venv; systemd-analyze --user verify of a rendered instance passes.
• ↳ codemod: deploy/systemd/orchestrata-codemod-daemon@.service: ExecStart=/opt/orchestrata-tools/bin/codemod-daemon /%I /%I/.codemod_daemon/socket /%I/.codemod_daemon/pid, WorkingDirectory=/opt/orchestrata-tools, the same ExecStartPre, keep Restart=on-failure/RestartSec=15, MemoryMax=4G (1.9 GB measured today). Installed by the same installer arm; the same unit-assertion test module covers both templates (parametrised over the two files).
• P3 build decisions (2026-10-06, reversible, recorded here per the decision skill): (1) deploy/systemd/install.sh already copies every deploy/systemd/*.service (templates included) into ~/.config/systemd/user with content compare + daemon-reload, but it is run only by prod promotion, which is refused until P5 — so install-tools.sh must put the two templates in place itself. "Reuse, do not duplicate" is met by giving install.sh an --only <glob> filter and having install-tools.sh call deploy/systemd/install.sh --only 'orchestrata-*-daemon@.service' from the staged tree; templates are skipped from enable/start there already. (2) A non-default PREFIX never touches ~/.config/systemd/user (the P2 rule for /usr/local/bin): the units go to $PREFIX/systemd-user and no daemon-reload runs. (3) The default-prefix path is a real-prefix-only path (P2 lesson), so the installer's verify arm asserts, by name, that the installed template is byte-identical to the staged one and that systemd-analyze --user verify passes on a rendered instance.
• P3 build record (2026-10-06): (a) codemod MemoryHigh lowered 4G → 3G with MemoryMax 6G → 4G, so the throttle point stays below the kill point (reversible operational pick; the old pair was 4G/6G). (b) Two defects fixed on the way, both found by the installer tests: systemd-analyze --user fails with "Failed to initialize manager" when XDG_RUNTIME_DIR is unset (cron/sudo/ssh), so the installer sets it to /run/user/$(id -u) for that call; and a filtered install.sh --only run no longer writes the converged-deployment stamp, because a partial run has not converged the services it skipped. (c) deploy/systemd is added to the installer's git archive list so the units come from the same pinned commit as the code. (d) Known gap: the --only tests run only --dry-run, because a real install.sh run ends in systemctl --user daemon-reload on the live user manager; the real-install branch of --only is exercised by the default-prefix path of every ship (named OK lines), not by a unit test. (e) Host finding, untouched: ~/.config/systemd/user still holds the plain orchestrata-ast-daemon.service / orchestrata-codemod-daemon.service / orchestrata-canvas.service files that make systemd-analyze print StartLimitIntervalSec noise — P5 removes them.
• P3 review round (2026-10-06, verdict HOLD → fixed): the reviewer caught three things the build and its tests had missed. (1) BLOCKER: the tracked plugin mirror plugins/orchestrata/systemd/ had drifted (ship's gen_plugin.py --check gate fails on drift) — regenerated. (2) The plan's own ExecStartPre=/usr/bin/test -d /%I/.git rejects every git worktree, where .git is a file; systemd-analyze verify never executes ExecStartPre, so the installer's verify arm cannot see it, and the unit test had encoded the bug by creating .git as a directory. Fixed to -e; the test now covers both shapes. (3) A non-default PREFIX install verified templates that still exec /opt/orchestrata-tools/bin/*, so it depended on the OLD real deployment (and fails on a host with none). The installed copy stays verbatim; the verified render is rewritten to $PREFIX/ first, and the unit test renders against a tmp tree of executable stubs. Lesson added to the P2 one: a verifier that does not execute a directive (verify vs ExecStartPre) proves syntax, not the precondition — the precondition needs a test of its own.
• P3 first ship (2026-10-06, 9c1cd1729): FAIL at the test-purity gate, before the suite ran (reports/run-tests-latest.json = {"aborted": "gates"}). The gate is lexical: the new unit test named its tmp launcher files stub_*, and "stub" is a forbidden-double token. The files are real executables that systemd-analyze verify checks for executability, not doubles; renamed to launcher_* and the docstring reworded. Lesson: the purity gate judges vocabulary as well as behaviour, so a test about placeholder files uses "placeholder"/"launcher", never stub/fake/mock/dummy.

P4 — hook and client use the deployed daemon only, verify identity

.claude/hooks/start-ast-daemon.sh (+ claude/hooks/ mirror via gen_plugin), scripts/serialize_codebase_ast.py

• Hook (two branches, per R1 — no spawn from a checkout venv under any condition): (1) if /health answers AND code_root starts with /opt/orchestrata-tools/ → done; (2) else if /opt/orchestrata-tools/bin/ast-daemon exists → stop any daemon on the socket whose code_root is a working tree (the hook REPLACES deploy-in-place daemons rather than tolerating them), then systemctl --user --runtime start orchestrata-ast-daemon@$(systemd-escape --path "$REPO_ROOT"); (3) no deployment → print one line (ast-daemon: /opt/orchestrata-tools not deployed; AST queries take the slow path — run deploy/install-tools.sh) and exit 0. The setsid spawn is deleted, not kept behind a flag.
• Client: unchanged protocol; query_ast_daemon logs (debug) code_root from /health once per process so a stale daemon is visible in --verbose.
• Tests: new tests/claude/test_start_ast_daemon_hook.py (no test covers this hook today; git ls-files tests | grep start_ast_daemon is empty) with the three branches, each with a REAL socket: a fixture daemon answering code_root=/opt/... → nothing started; code_root=/home/... → stop + start path taken (systemctl presence is a test seam via PATH); no deployment → the one-line message, exit 0, no process spawned (assert via the fixture's process table before/after).
• ↳ codemod: CodemodDaemon.start(repo_root) stops spawning sys.executable -m …. It starts orchestrata-codemod-daemon@<escaped repo_root> through systemctl --user --runtime start when /opt/orchestrata-tools/bin/codemod-daemon exists, and raises a clear error naming deploy/install-tools.sh otherwise; stop and status go through the unit too (status keeps reading the pid file and now also reports identity). scripts/codemod.py surfaces that error verbatim. Tests in tests/refactor/codemod/test_daemon_lifecycle.py switch to a real systemctl --user against a runtime-installed template pointing at a tmp deployment prefix (the unit's ExecStart path comes from the installer's PREFIX seam added in P2), and one test proves the no-deployment error.

P4 pre-build record (2026-10-06, measured in wt-AST-DAEMON-P4 at d46b99595):

• False premise corrected — "no test covers this hook today". It is covered: tests/scripts/test_ast_daemon.py:642-763 holds five hook tests (test_hook_bash_syntax_ok, test_hook_cold_start_spawns_a_healthy_daemon, test_hook_fast_path_skips_spawn_when_already_healthy, test_hook_idempotent_under_concurrent_cold_start, test_hook_spawns_via_venv_python_not_uv_run). Three of them pin the setsid venv spawn that P4 deletes; they are rewritten into the new tests/claude/test_start_ast_daemon_hook.py against the three P4 branches, not kept and not silently dropped (the git ls-files | grep start_ast_daemon probe in the packet text matched on file NAME only).
• False premise corrected — "scripts/codemod.py surfaces that error verbatim". scripts/codemod.py parses --auto-start and --no-daemon but never reads either; CodemodDaemon.start has ZERO production callers (only its two test modules). P4 therefore wires --auto-start to CodemodDaemon.status/start as the first production caller (RUL-023) so the sentence becomes true; the no-deployment RuntimeError is printed verbatim to stderr, exit 1.
• Operational picks (reversible, recorded here per the decision skill): (a) systemctl --user start without --runtime — systemctl documents --runtime for enable/disable/edit only; a start is never persisted, so the flag in the packet text was a no-op. (b) Test seams are identity-shaped, not behaviour-shaped: the hook reads AST_DAEMON_TOOLS_PREFIX (default /opt/orchestrata-tools) and AST_DAEMON_UNIT (default orchestrata-ast-daemon); CodemodDaemon takes tools_prefix= and unit_name= constructor kwargs with the same defaults. Tests install a DISTINCTLY NAMED runtime template under $XDG_RUNTIME_DIR/systemd/user/ (derived from the repo template by the installer's own s#/opt/orchestrata-tools/#<prefix>/#g rewrite) pointing at a tmp deployment prefix whose bin/* launchers exec this checkout's .venv interpreter — a real unit, a real daemon, a real socket, and no venv rebuild. Distinct names are load-bearing: ~/.config/systemd/user (where P3 installed the real templates) takes precedence over the runtime unit dir, so a same-named test template would be shadowed. (c) The codemod daemon entry does NOT write its pid file (its docstring: "written by parent"); under systemd CodemodDaemon.start writes MainPID read from systemctl --user show after the start returns. (d) The deployed AST daemon's code_root is /opt/orchestrata-tools/lib (launcher runs lib/scripts/ast_daemon.py); the hook's identity test is "code_root starts with $AST_DAEMON_TOOLS_PREFIX/", which that satisfies.

P4 build record (2026-10-06, builder commits 3bc89cca5 + 2ae7a8090):

• Built as specified: hook three-branch (no venv spawn left; mirror regenerated, gen_plugin.py --check current), client identity DEBUG log once per process, CodemodDaemon.start/stop/status through systemctl --user with the deployment check first, scripts/codemod.py --auto-start wired. New shared helpers tests/fixtures/deployed_daemons.py (runtime template installer + tmp deployment prefix); new tests/claude/test_start_ast_daemon_hook.py (7 tests, all branches with a real unit/daemon/socket). Builder-run counts: hook 7, ast_daemon 42, daemon_lifecycle 42, daemon_coverage 5, codemod_cli 43, gen_plugin 35 — all passed. ruff/black/pyright/gate.py/check-deps/purity all clean.
• Builder deviations, accepted (operational): (1) status()["identity"]["deployed_commit"] is the sha parsed from the VERSION commit <sha> line (same as the daemons' own /health), not the raw first line. (2) The hook's replace branch waits (≤3 s) for the old daemon's socket AND pid to disappear before systemctl start, otherwise the new daemon loses the .lock race and sits out RestartSec=10. (3) scripts/codemod.py gained CODEMOD_TOOLS_PREFIX / CODEMOD_DAEMON_UNIT env seams (via os.environ.get) plus kwargs on _auto_start_daemon, because /opt/orchestrata-tools IS deployed on this host and the no-deployment test needs another prefix; main() returns 1 (the __main__ guard does sys.exit(main())). (4) The test template rewrite also rewrites the bare WorkingDirectory=/opt/orchestrata-tools line, which the installer's s#/opt/orchestrata-tools/#…#g (install-tools.sh:598) does not. Checked 2026-10-06: that rewrite feeds ONLY the systemd-analyze --user verify render for a non-default PREFIX; the installed copy is always the verbatim template, and systemd-analyze verify does not require WorkingDirectory to exist — so no installed unit is affected. The test helper needs the extra rewrite because its unit is actually STARTED from the tmp prefix. Not a defect; a note for whoever extends the installer's PREFIX seam to real installs. (5) The four obsolete spawn tests in tests/scripts/test_ast_daemon.py are replaced by one lexical pin (test_hook_source_has_no_checkout_spawn); the behavioural branch tests live in the new file.
• Unrelated findings (RUL-024, not fixed in P4): tests/scripts/test_ast_daemon.py::test_foreign_cwd_defaults_land_under_repo_and_watcher_and_check_deps_work timed out twice on /check-deps (5 s client timeout) under load and passed on re-run with and without the P4 patch — load-dependent, not P4. .claude/skills/ast-toolkit/SKILL.md:171-174 still describes the venv spawn → P7 docs. docs/adr/0067-…md:28 describes the pre-P4 hook as measured context; an Accepted ADR body is immutable, so that line stays as history.

P4 review round 1 (2026-10-06, reviewer on 3bc89cca5+2ae7a8090): HOLD.

• BLOCKER 1 (reproduced, rc=143): tests/claude/test_start_ast_daemon_hook.py builds the hook env from os.environ; tests/scripts/conftest.py's session-scoped session_ast_daemon exports AST_DAEMON_SOCKET for an in-process daemon whose /health pid IS the pytest process. When any tests/scripts test runs first, the hook's replace branch SIGTERMs pytest. Fix: the test pins AST_DAEMON_SOCKET/RUN_DIR/LOG to its own world; the hook refuses to kill a pid that is $$/$PPID or whose /proc/<pid>/cmdline does not name ast_daemon.
• BLOCKER 2 (reproduced): scripts/check_external_tools.py → UNDECLARED systemd-escape (daemon.py:77) and systemctl (daemon.py:89). The build record's "gates clean" claim was wrong for this gate. Fix: declare both under [tool.orchestrata.external-tools].
• MINOR 3-8 (hook: tool/template checks before the kill; a daemon answering without identity is tolerated → lock-race churn; identity parse splits on whitespace; codemod pid file stale after a systemd restart → stop may signal a reused pid; _unit_instance crashes on missing systemd-escape; the client identity log has no --verbose to surface it). All fixed in round 2.
• Design note 9 (not P4): --auto-start starts but does not connect (REQ-110-140 "auto-connect") — SPEC-110 scope beyond ADR-0067, recorded as a follow-up, not built under the waiver.
• Pre-existing in test_daemon_lifecycle.py (RUL-024): test_start_creates_daemon_dir never calls start(); test_stop_sigkill_on_timeout reassigns the module constant SHUTDOWN_TIMEOUT (a monkeypatch the lexical gate misses). Both fixed in round 2 since the file is being rewritten anyway.
• coverage-claude gate timed out at 300 s under a loaded machine (another worktree's suite + the reviewer's own runs) — needs a clean rerun; not attributed to P4.

P4 fix round 2 (2026-10-06, builder, commit 46e2d0a4f): both blockers, minors 3-8 and both pre-existing test defects addressed; awaiting reviewer re-check. Builder-reported evidence (orchestrator has not yet re-run it):

• Blocker 1 reproduction tests/scripts/test_ast_daemon.py::test_hook_source_has_no_checkout_spawn tests/claude/test_start_ast_daemon_hook.py in ONE pytest process, scripts test first (session daemon env live): rc=0, 12 passed, 44.80 s. The test world now pins AST_DAEMON_SOCKET/RUN_DIR/LOG; the hook refuses $$/$PPID and any pid whose /proc/<pid>/cmdline lacks ast_daemon (stderr ast-daemon: refusing to signal pid <pid> (not a separate ast_daemon process); not replacing), proven by a self-pid test (real /health unix socket answering pid=os.getpid()) and a bystander sleep pid test. A builder rc=143 on the first attempt was its own timeout 300 under load ~21, not the hook; the passing run used timeout 590.
• Blocker 2: systemctl and systemd-escape declared under [tool.orchestrata.external-tools] (min_version 250.0, install hint); check_external_tools.py → OK systemctl 257.5 >= 250.0, OK systemd-escape 257.5 >= 250.0, PASS no undeclared invocations.
• Minor 3: pre-flight systemctl --user cat "<unit>@.service" before any kill; stderr ast-daemon: cannot start <unit> (systemctl/systemd-escape missing or template not loadable); leaving any running daemon alone. Minor 4: identity-less socket → stdout ast-daemon: socket answers without identity; not replacing — restart it by hand, exit 0. Minor 5: identity fields tab-separated, read with IFS=$'\t'; self-pid test uses a code_root with spaces. Minor 6: status() takes pid/running from the unit's MainPID when loaded, pid file only when not; stop() on a loaded unit is systemctl stop + file cleanup, never signals the pid-file pid, failed stop → RuntimeError; legacy SIGTERM path kept for an unloaded unit. Minor 7: _unit_instance → RuntimeError on missing/failing systemd-escape (empty-PATH child test). Minor 8: serialize_codebase_ast.py --verbose (global) → logging.basicConfig(DEBUG, stderr); subprocess test shows exactly one identity line with it, none without.
• Test defects: test_start_creates_daemon_dir now calls start(); CodemodDaemon takes a shutdown_timeout kwarg and both SIGKILL tests pass 0.5 instead of reassigning the module constant; the lifecycle SIGKILL test asserts death + cleanup (stop's liveness probe can reap its own child → exit 0), the -9 assertion stays in test_daemon_coverage.py. The old "failed unit start" hook test used a nonexistent unit (now caught by pre-flight); it removes .git so the unit loads and ExecStartPre fails.
• Per-module (builder logs): hook module 11 passed (160 s under load); test_ast_daemon.py 43 passed; test_daemon_lifecycle.py 45 passed; test_daemon_coverage.py 5 passed; test_codemod_cli.py 43 passed; test_gen_plugin.py 35 passed. Caveat the builder stated: hook, coverage, codemod-CLI and gen-plugin modules ran BEFORE its last test-only edit (one assertion in test_stop_sigkill_on_timeout); only lifecycle, test_ast_daemon.py and the reproduction ran after it. bash -n hook OK, gen_plugin.py --check current, ruff/black clean, pyright 0 on daemon.py, no p4-* units or p4h-* scratch dirs left behind.
• Still not built (correct under the ADR-0048 waiver): REQ-110-140 auto-connect; --auto-start is start-only. Owned by SPEC-110 packet S4 (plans/spec-110-completion-plan.md).

P4 review round 2 (2026-10-06, reviewer on 46e2d0a4f): HOLD. Both round-1 blockers reproduced closed: the ordered pair test_ast_daemon.py::test_hook_source_has_no_checkout_spawn + tests/claude/test_start_ast_daemon_hook.py in ONE invocation → 12 passed in 79.15s, rc=0, pytest not signalled (hook refuse branch at start-ast-daemon.sh:102-107 covers $$, $PPID, cmdline-without-ast_daemon; self-pid and bystander tests drive a real ThreadingUnixStreamServer /health); check_external_tools.py → 8 declared, systemctl/systemd-escape OK 257.5 ≥ 250.0, PASS. Minors 3, 4, 5, 6 (loaded-unit half), 8 proven by execution; minor 7's CalledProcessError branch proven by reading only. Test defects fixed; shutdown_timeout default == old constant 5.0. Gates: bash -n OK both hook copies (byte-identical), gen_plugin.py --check current, ruff/black clean on the 6 .py files, no attribution, no leftover p4-* units or scratch dirs. Six modules on the committed tree: hook 11 passed (126 s); lifecycle 45 passed; coverage 5 passed; codemod CLI 43 passed; gen_plugin 35 passed; tests/scripts/test_ast_daemon.py RED in 2 of 2 runs (run 1: 2 failed, 41 passed, 1 error, load 38.9; run 2: 1 failed, 42 passed) — the builder's "43 passed" did not reproduce. HOLD reasons, all owned by fix round 3:

• N1 (minor): the hook's identity check is a cmdline SUBSTRING match, so the pytest hosting conftest's in-process daemon (argv contains tests/scripts/test_ast_daemon.py) passes it; only $PPID protects pytest, and any intermediate process (sh -c, timeout, _dispatch.sh's timeout 120 bash) defeats $PPID. Fix: match scripts/ast_daemon.py as an argv ELEMENT, or compare the pid against the unit's MainPID; add the test.
• N2 (minor, minor-6 residual): CodemodDaemon.stop() on an UNLOADED unit still SIGTERM/SIGKILLs the stale-pid-file pid with no cmdline check (daemon.py:243-250); probe scratchpad/p4r2_probe/legacy_stop_probe2.py killed an unrelated setsid sleep 60. Fix: cmdline check (same predicate as the hook) before any signal on the legacy path.
• N3 (minor): loaded-but-inactive unit + stale pid file pointing at an unrelated live pid → status() says not running, start() (daemon.py:162-167) refuses "already running" → --auto-start wedges until someone deletes the file by hand (probe autostart_probe.py). Fix: start() applies the same unit-first rule as status(); a stale pid file under a loaded unit is removed, not trusted.
• N4 (ship gate): test_ast_daemon.py red — test_foreign_cwd_defaults_land_under_repo_and_watcher_and_check_deps_work (httpx.ReadTimeout on /check-deps, 5 s client timeout) failed in both runs and passed alone (47.91 s); test_toolkit_get_routes_wait_on_cache_lock_but_health_does_not failed once (0.64 s vs < 0.5 s at load 39). Neither test nor scripts/ast_daemon.py is in the P4 diff. Attribution is not yet proven either way: the module must be green on the committed tree under observed load, or the two tests get a measured, recorded load-tolerance fix (RUL-006, fix on sight), never a skip.
• N7 (DoD, ADR-0069): no @traces on _unit_loaded or on any test added in 46e2d0a4f; RTM, docs, wiki, coverage items unverified.
• Notes, not blocking: N5 lifecycle SIGKILL test has no readiness handshake (coverage test does); N6 self-pid test cannot tell which check refused; N8 eight leaked P2-era /tmp/p2t-*/.../ast_daemon.py processes alive (pids 1088290–1303767) — host cleanup, not P4 code.
• Routed to Mohamed, not P4: the AST-daemon hook is registered at PROJECT level (.claude/settings.json:26), and the user-level plugin dispatcher plugins/orchestrata/project-hooks/_dispatch.sh exits when a project .claude/settings.json exists; both copies are byte-identical and the registration pre-dates P4 (on base 83c604886). Whether hooks live at user level only is a standing-rule question for Mohamed (P7 scope at the earliest).

P4 fix round 3 (2026-10-06, on top of 13e551b46). DESIGN_WAIVER: ADR-0067.

• N1 closed: start-ast-daemon.sh (both copies, byte-identical) replaces the cmdline substring match with an argv-ELEMENT match (is_ast_daemon_argv: an element that is ast_daemon.py or ends in /ast_daemon.py; the deployed launcher execs python <prefix>/lib/scripts/ast_daemon.py). The ".../scripts/" prefix is not spelled in the hook because test_hook_source_has_no_checkout_spawn pins that literal out of the hook source. A pid is also accepted when it equals the loaded unit's MainPID; a pid that is neither, or is $$/$PPID, is refused. Test: argv[0] python -m pytest tests/scripts/test_ast_daemon.py (exec -a ... sleep) is refused and survives.
• N2 closed: _pid_is_codemod_daemon(pid) (argv element orchestrata_ultimate. refactor.codemod._daemon_entry directly after -m) gates the legacy (unit-unloaded) path of CodemodDaemon.stop(): a failing pid is reported, the stale files removed, nothing signalled. Test: non-child setsid sleep survives.
• N3 closed: start() runs _refuse_if_running: unit loaded -> MainPID decides and a stale pid file is removed; unit unloaded -> the pid file counts only for a pid that passes the predicate. Probes legacy_stop_probe2.py (unrelated pid ALIVE after stop) and autostart_probe.py (no "already running"; start and status agree) re-run against the old and new code.
• N4 attributed, not assumed: scripts/ast_daemon.py is unchanged vs 83c604886. Under load average ~40 a cold daemon needed 34.9 s to answer /health, /check-deps took 21.9 s cold and 6.2-8.3 s warm, and /health+/healthz with the cache lock held showed a worst honest latency of 0.907 s (120 requests). So the tests, not the daemon, were load-bound: the /check-deps client timeout is now 60 s, the lock-wait bounds 5 s (a request waiting on the held lock never returns inside the block), _HEALTH_POLL_TIMEOUT 90 s; each with its measurement in a comment. Baseline on the committed tree: run 1 43 passed (load 17); run 2 hit the 590 s timeout at load 44 after the foreign-cwd failure. After the fix: 43 passed twice (load 9 -> 29 in 159 s; 29 -> 51 in 298 s).
• N7 (ADR-0069): @traces(spec="110", ...) on _unit_instance, _unit_loaded, _systemctl, _unit_property, _wait_for_main_pid, _refuse_if_running, _remove_stale_pid_file, _pid_is_codemod_daemon and on every test added in 46e2d0a4f and this round (REQ-110-134/135, the ids the spec defines for the PID file and lifecycle); hook tests carry an ADR-0067 docstring line instead (no spec id exists for the hook). reports/rtm-evidence.{json,md} regenerated. trace_intel untraced-reqs: SPEC-110 untraced 19 before and after (REQ-110-133..139 were already traced; none of the 19 is in this packet's scope). daemon.py coverage 91% (264 stmts, 23 missed: systemd-escape error branches 88-94, systemctl rc 127 116-117, MainPID timeout 136-142, stale-file unlink OSError 185-186, and small defensive branches). docs/ast-toolkit.md has no hook section (P7 owns it), so nothing to update there.

P4 fix round 4 (2026-10-06, on top of ffadad98c). DESIGN_WAIVER: ADR-0067. Clears the round-3 reviewer HOLD items that need no ruling.

1. RTM blocker (d4fee128f): reports/rtm-evidence.{json,md} regenerated with scripts/generate_rtm_evidence.py --verify --verify-timeout 600 (178 citing modules, 33 min). Header: "Execution evidence this run: YES (--verify)". Totals: 7262 requirements, TESTED=333, FAILING=11, DECLARED=2974, UNLINKED=3944, FIXTURE=5, GHOST=4, MALFORMED=6. Eight modules graded RED in that run, none in the P4 diff: tests/conftest.py, tests/e2e/test_full_pipeline.py, tests/knowledge_graph/test_context_pack_performance.py, tests/orchestrator/test_pipeline_e2e_{analysis,docs_session,review}.py, tests/refactor/codemod/test_workspace_load.py, tests/scripts/test_trace_intel.py; not investigated this round.
2. TOCTOU (dccff2c00): CodemodDaemon.start() acquires the lock first and runs _refuse_if_running plus the launcher check inside the locked region; no systemctl start precedes either. Tests: test_start_refuses_only_under_the_lock (lock held by the test: start fails on the lock and the stale PID file the refusal would remove is still there) and test_concurrent_starts_exactly_one_proceeds (two threads, barrier, real unit: exactly one starts).
3. Repo-tied predicate (dccff2c00): _pid_is_codemod_daemon(pid, repo_root) requires argv -m orchestrata_ultimate.refactor.codemod._daemon_entry <repo_root> with the resolved repo root as the element right after the module (the launcher codemod-daemon <repo_root> <socket> <pid_file> and the unit's ExecStart ... /%I ... both pass it first). _refuse_if_running and stop() pass the repo root. TestPidIsCodemodDaemon gains the different-repo-root case and a -m <module> with no root case.
4. Tests (dccff2c00): hook MainPID branch — test_hook_accepts_the_units_main_pid_without_an_ast_daemon_argv and test_hook_refuses_a_pid_that_is_neither_the_main_pid_nor_an_ast_daemon against a real running user unit whose launcher is rewritten to a non-ast_daemon.py argv; lifecycle — test_the_real_units_main_pid_is_our_daemon_and_this_process_is_not against a real started unit. The hook body is unchanged (no defect exposed). Also test_zombie_child_counts_as_dead got a 60 s bound (was 5 s): it failed at load average ~35 on interpreter start-up alone, with the assertion unchanged.
5. Wiki (knowledge-wiki a84c634): claude/hooks/start-ast-daemon.md rewritten for the systemd user-unit hook, MainPID identity, argv matching, $$/$PPID refusal, no checkout spawn; the code block equals the committed hook byte for byte. Not pushed. Measured: daemon.py coverage 91% (268 stmts, 23 missed) over test_daemon_lifecycle.py + test_daemon_coverage.py (61 passed); hook module 14 passed. REQ-110-135's text amendment is ADR-0071 (Proposed, awaiting Mohamed); specs/110-codemod-engine/spec.md and its checklist are untouched.

P4 review round 4 (2026-10-06): GO. Reviewer re-measured 61 + 14 passed, daemon.py 91%, ruff/black clean, specs diff empty, wiki code block identical to the hook. The 8 RED RTM modules are none of them P4-caused: 5 are FAILING on master's RTM too (test_full_pipeline, the three test_pipeline_e2e_*, test_context_pack_performance), tests/conftest.py is a harness artefact (run_citing_tests runs pytest tests/conftest.py, exit 5 = nothing collected, graded RED), test_workspace_load.py is load-only (passes at idle; REQ-110-001..004 read FAILING in the committed RTM for that reason), test_trace_intel.py is a worktree-environment gap (.cache/trace_intel/codebase_facts.pl has 0 runtime_call( facts here, 7628 in the main checkout; cites only FIXTURE ids). Three MINOR findings carried to P5 as debt, none blocking:

• test_concurrent_starts_exactly_one_proceeds accepts either "Cannot acquire lock" or "already running", so it passes on the pre-fix ordering too; the discriminating proof is test_start_refuses_only_under_the_lock. Narrow or rename it.
• No test proves the lock is released when the refusal raises inside the locked region (the finally does release it); add a re-acquire assertion after pytest.raises(..., match="already running").
• _pid_is_codemod_daemon returns False for a RELATIVE repo-root argv (Path(".").resolve() resolves against the caller's cwd); unreachable from the deployed launcher, template and start() (all absolute), reachable only by a hand-started legacy daemon. Resolve against /proc/<pid>/cwd when the argv path is relative. Also owed, separate owner: run_citing_tests grading a conftest.py citation as a module (exit 5 → RED); an idle-load --verify regeneration before the next RTM commit so REQ-110-001..004 read TESTED.

P5 — retire the checkout-bound units and gate their return

• Host action (part of the P5 ship's verify, not a hand step): systemctl --user disable --now and remove the file for BOTH orchestrata-ast-daemon.service and orchestrata-codemod-daemon.service (R2). Before the codemod unit stops, confirm no client holds its socket (ss -xp | grep .codemod_daemon); a held socket is reported, not raced.
• Gate: scripts/check_prod_tree_separation.py (RUL-027) gains a check over ~/.config/systemd/user/*.service: any non-template unit whose ExecStart or WorkingDirectory matches /home/*/workspace/ is a violation. Test in tests/scripts/test_check_prod_tree_separation.py with fixture unit dirs.
• deploy/install-tools.sh verify: neither plain unit may exist (systemctl --user cat <unit> fails for both) — a deploy that leaves an old unit in place is not finished.

P5 build record (2026-10-06, builder commit 240053176 on feat/ast-daemon-p5-retire-units, DESIGN_WAIVER ADR-0067 R1/R2).

Host before-measurement (builder session, before any host action): orchestrata-ast-daemon.service and orchestrata-codemod-daemon.service both enabled / inactive under ~/.config/systemd/user; ss -xp showed no client on any .codemod_daemon/socket. Real-host python scripts/check_prod_tree_separation.py (default unit dir), exit 1 — 20 units examined, 4 offending directives in the 2 plain units, none yet retired (the ship retires them, never a hand step):

[prod-separation] FAIL — RUL-027: 20 unit(s) examined, 4 directive(s) point production at a DEVELOPMENT checkout (--dev-tree <this worktree>)
  orchestrata-ast-daemon.service:15: WorkingDirectory=/home/mohamed/workspace/orchestrata-ultimate-architecture  [under a /home/*/workspace/ checkout (non-template unit)]
  orchestrata-ast-daemon.service:17: ExecStart=/home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \  [under a /home/*/workspace/ checkout (non-template unit)]
  orchestrata-codemod-daemon.service:14: WorkingDirectory=/home/mohamed/workspace/orchestrata-ultimate-architecture  [under a /home/*/workspace/ checkout (non-template unit)]
  orchestrata-codemod-daemon.service:16: ExecStart=/home/mohamed/workspace/orchestrata-ultimate-architecture/.venv/bin/python \  [under a /home/*/workspace/ checkout (non-template unit)]

What shipped:

• Gate (scripts/check_prod_tree_separation.py): a second rule next to --dev-tree. Any NON-template unit (@ not in the unit name) whose ExecStart, ExecStartPre, ExecStartPost or WorkingDirectory carries a path that, after os.path.abspath (no symlink resolution), has parts[1] == "home" and parts[3] == "workspace" is a violation with the reason under a /home/*/workspace/ checkout (non-template unit); the --dev-tree rule keeps its own reason under the --dev-tree. Both can fire on one line (reported once per reason; find_violations still counts the line once). Template and instance units are exempt from the workspace rule only. Judgment call (the plan allowed it): the rule covers those four exec/cwd directives, NOT every PATH_DIRECTIVE (Environment=, EnvironmentFile=, StandardOutput= stay under the --dev-tree rule alone), because only exec/cwd decide what a unit runs from; widening is a one-line change to WORKSPACE_DIRECTIVES if Mohamed wants it. identity_core's DEFAULT_GLOBS / PATH_DIRECTIVES are untouched.
• Retire + verify (deploy/retire-plain-units.sh, new; called by deploy/install-tools.sh on the DEFAULT prefix only): retire refuses BEFORE touching anything while ss -xp shows a client connected to a <repo>/.codemod_daemon/socket named in a plain unit (reports pid=<n> <cmdline>, kills nothing), else systemctl --user disable --now, removes the file, daemon-reload; a second run is a no-op. verify fails while systemctl --user cat <unit> succeeds for either unit. The installer calls retire in the default-prefix branch of "systemd templates" and verify after the template checks, before mv "$VERSION_STAGING" "$PREFIX/VERSION"; a tmp PREFIX never reaches either. The logic lives in its own script (installed from the staged commit like deploy/systemd/install.sh) because the installer cannot run with the default PREFIX from a test, while the script takes UNIT_DEST / PLAIN_UNITS and runs for real against uniquely named units.
• P4 MINORs closed (all in daemon.py + test_daemon_lifecycle.py):
  1. test_concurrent_starts_exactly_one_proceeds renamed test_concurrent_starts_launch_the_unit_once_and_refuse_the_other, docstring says it is the race smoke and points at test_start_refuses_only_under_the_lock as the discriminating proof. Narrowing to "already running" was rejected: the loser legitimately gets the lock refusal when the winner still holds the lock, so it would flake.
  2. _assert_lock_reacquirable after pytest.raises(..., match="already running") in both refusal tests.
  3. _pid_is_codemod_daemon resolves a relative argv repo root against os.readlink("/proc/<pid>/cwd"); an unreadable cwd link is the same fail-safe False. Tests: relative root from a different cwd (True for the served dir, False for a decoy and a nonexistent sibling) and a non-dumpable (prctl(PR_SET_DUMPABLE, 0)) daemon-shaped child whose cwd link raises EACCES.
  4. _await_argv (wait_until: cmdline non-empty and holding -m + the entry module) before every argv assertion in TestPidIsCodemodDaemon and test_start_raises_when_already_running. Production predicate unchanged.

Tests shown to fail on the pre-fix code (no git stash): (3) git checkout afd1e82ae -- daemon.py (pre-fix predicate) then the class: 2 failed (the relative-root test and the unreadable-cwd test), 5 passed; restored with git checkout HEAD --. (2) release removed from the finally (pass): both already running tests failed (Lock acquisition failed: [Errno 11]); restored. (4) a 400-spawn loop under 16 busy threads: the old no-wait shape read the real daemon-shaped child as not-a-daemon 36 / 400 times, the _await_argv shape 0 / 400. (1) is a rename, not a strengthened assertion.

Measured: 242 passed, 0 skipped in one run over test_check_prod_tree_separation.py (42 collected), _paths, test_daemon_lifecycle.py (58) + test_daemon_coverage.py (5) and tests/promotion/test_identity_*.py; 33 passed, 0 skipped over tests/deploy/test_install_tools.py + test_ast_daemon_unit.py + test_retire_plain_units.py. Coverage from the first run: scripts/check_prod_tree_separation.py 96 % (89 stmts, 4 missed: 110-111, 154-155 — the OSError guards and the no-unit-dir early return), refactor/codemod/daemon.py 91 % (269 stmts, 23 missed, all pre-P5 branches). RTM: no @traces set changed (_pid_is_codemod_daemon keeps REQ-110-135; the new tests carry @traces(spec="110", reqs=["REQ-110-135"]) on the same id), so the committed RTM is left alone.

Ship acts (not done by the builder): the P5 ship's installer run performs the retirement. Expected after it: systemctl --user cat fails for both plain units, ~/.config/systemd/user/orchestrata-{ast,codemod}-daemon.service absent, and the real-host check_prod_tree_separation.py exits 0.

Review round 1 (NO-GO) → fix round 1 (2026-10-06, DESIGN_WAIVER ADR-0067: every item hardens P5 deliverables (b)/(c) or closes a P4 MINOR more precisely).

• MAJOR-1 — workspace rule missed ordinary spellings (scripts/check_prod_tree_separation.py). Fix: (1) _logical_lines joins physical lines ending in \ into one logical directive before partition("="), reporting the FIRST physical line's number; (2) _workspace_candidates (workspace rule only) strips single/double quotes from every token, the exec prefixes @ - : + ! | from the first token, and expands %h to Path.home() (%u/%U are NOT expanded: they never stand in for a /home/<user>/workspace prefix on their own); (3) WORKSPACE_DIRECTIVES gains ExecStop, ExecStopPost, ExecReload, ExecCondition — none is in identity_core.PATH_DIRECTIVES, so the directive filter now admits PATH_DIRECTIVES | WORKSPACE_DIRECTIVES while the --dev-tree rule stays scoped to PATH_DIRECTIVES (its behaviour and identity_core are unchanged); (4) the FAIL trailer is per reason: a workspace hit says "A plain unit points at a /home//workspace/ checkout", the "While a prod unit points here …" text only appears when a --dev-tree hit exists. 23 new collected tests, one fixture per spelling (%h WorkingDirectory + ExecStart, \ continuation + first-line number + next-directive-not-swallowed + unterminated tail, @ prefix, every prefix, double- and single-quoted executable, the four Exec directives, clean controls, both trailers). Pre-fix evidence (git checkout 49cb6c600 -- <script>, the new tests -k-selected, restored from a saved copy — never git stash): 15 failed, 8 passed (the 8 passes are the clean controls and the single-character prefixes - : + ! that _candidate_paths already stripped); post-fix 70 passed over test_check_prod_tree_separation.py + _paths.
• MINOR-1 — retire test never retired an ENABLED unit (tests/deploy/test_retire_plain_units.py). Fixture units now carry [Install] WantedBy=default.target; test_retire_disables_an_enabled_unit_and_removes_its_wants_symlink runs systemctl --user enable --runtime (links under $XDG_RUNTIME_DIR, isolated), asserts enabled-runtime + the default.target.wants/<unit> symlink before, and not-enabled + symlink gone after retire. It exposed a real defect: disable --now acts on persistent enablement only, so a runtime enablement left a dangling symlink — retire now also runs disable --runtime (best effort). Fixture teardown also disables --runtime and unlinks the wants symlink.
• MINOR-2 — retire/verify trusted the file on disk (deploy/retire-plain-units.sh). unit_is_live reads systemctl --user show -p LoadState,ActiveState,UnitFileState (live = not not-found, or not inactive, or a unit-file state; an unreachable manager reads live, fail closed). With the file gone and the unit live, retire runs stop + best-effort disable (disable cannot act on a missing file), then the existing daemon-reload; with the file gone and the unit not live it is "already retired". verify additionally fails while is-active is active|activating|reloading|deactivating. Tests: file gone + units running → verify fails, retire stops (no "already retired"), verify then passes; and an is-active-arm isolation test. Pre-fix evidence (old script, same -k selection): 3 failed.
• MINOR-3 — _pid_is_codemod_daemon read the cwd link unconditionally (refactor/codemod/daemon.py). /proc/<pid>/cwd is now read only when the argv repo root is NOT absolute; an absolute root with an unreadable cwd stays identified. test_an_unreadable_cwd_link_is_fail_safe_not_a_daemon now points at a RELATIVE root; new test_an_absolute_root_is_identified_without_the_cwd_link (True for the served root, False for another) shares a _non_dumpable_daemon helper. @traces REQ-110-135 unchanged. Pre-fix evidence: the absolute-root test fails on the old predicate (the relative-root one passes on both, as the old code was already fail-safe there).
• MINOR-4 — docstring overclaim (test_concurrent_starts_launch_the_unit_once_and_refuse_the_other): rewritten to what it proves (exactly one start() returned, the other raised, the unit ends with a live MainPID) and names test_start_refuses_only_under_the_lock as the discriminating proof. Docstring only.
• Reviewer NOTEs NOT changed, with reasons. Template exemption by unit NAME only (@ in the name): brief-specified. Symlinks not followed by the workspace rule: brief-specified (the unit file's text is the evidence). socket_clients reports the first pid only: the refusal still fires, which is the contract. The AST daemon socket is not checked by retire: the plan requires the codemod-socket check only. A missing ss fails closed through the installer's fail. Retirement is not rolled back by a later installer failure: the plan accepts it (a retired plain unit has no reason to return; the templates replace it).

Measured (fix round 1): 70 passed over test_check_prod_tree_separation.py + _paths; 64 passed over test_daemon_lifecycle.py (59) + test_daemon_coverage.py (5); 36 passed over test_install_tools.py + test_ast_daemon_unit.py + test_retire_plain_units.py (12); 254 passed over tests/promotion/; 424 passed, 0 skipped in total. Coverage (superseding the first-run figures above): scripts/check_prod_tree_separation.py 97 % (119 stmts, 4 missed: 166-167, 210-211 — both pairs are OSError guards: the _is_under_root resolve() guard and the unit read_text guard; the no-unit-dir early return is covered), refactor/codemod/daemon.py 92 % (274 stmts, 23 missed, all pre-P5 branches). Real-host scripts/check_prod_tree_separation.py is still exit 1, same 4 directives (before-state, retired by the ship); the two ExecStart lines now print as one logical line (the \ continuations joined) at their first physical line, 17 and 16.

Review round 2 (GO with MINORs) → fix round 2 (2026-10-06, DESIGN_WAIVER ADR-0067: every item hardens deliverable (b) gate / (c) retire-verify or fixes a test race; no new design decision).

• M1 — escaped backslash read as a continuation (scripts/check_prod_tree_separation.py). _logical_lines now continues only on an ODD count of trailing backslashes ((len(line) - len(line.rstrip("\\"))) % 2 == 1), as systemd does. Pre-fix evidence (old script via git checkout HEAD -- <script> at 872ce77d8, the new copy restored from a saved file, never git stash): SyslogIdentifier=x\\ followed by WorkingDirectory=/home/u/workspace/r gave [] (swallowed); post-fix it reports line 3 under the workspace reason. Tests: test_an_escaped_trailing_backslash_is_not_a_continuation, test_three_trailing_backslashes_still_continue.
• M2 — a comment line inside a continuation ended the buffer. With the buffer non-empty a line starting # or ; is skipped without ending it (systemd joins across it). Pre-fix: ExecStart=/usr/bin/env \ / # note / /home/u/workspace/r/run gave []; post-fix line 2 under the workspace reason. Test: test_a_comment_line_inside_a_continuation_is_skipped_not_terminating (# and ;).
• M3 — %h tests depended on the host's home. monkeypatch is forbidden, so the seam is an injection: _workspace_candidates(value, home=None) (default Path.home()), threaded through a keyword-only home on find_violations_with_reasons and find_violations. The two %h tests pass home=Path("/home/u"); the new test_percent_h_under_a_home_outside_home_star_is_not_a_violation passes home=Path("/root") and asserts %h/workspace/r is NOT a violation, recording that ADR-0067 decision 4 defines the violation literally as /home/*/workspace/. Pre-fix the three tests fail (the gate has no home parameter).
• M4 — a unit left failed was retired on every run (deploy/retire-plain-units.sh). reset-failed follows the stop in both arms, and unit_is_live no longer counts LoadState=not-found + ActiveState=failed + empty UnitFileState as live. Tests: test_a_unit_left_failed_is_retired_once_and_the_second_run_is_a_noop (runtime unit ExecStart=/bin/false, started, retired twice; the second run prints "already retired", no daemon-reload) and test_a_failed_unit_whose_file_is_gone_is_not_live. Pre-fix: both fail on the old script.
• M5 — verify passed with the manager unreachable. verify now fails when the unit file (or symlink) exists in UNIT_DEST, and when systemctl --user show -p LoadState itself exits non-zero ("cannot query the systemd user manager ..."). Tests: test_verify_fails_when_the_unit_file_is_present_and_the_manager_is_unreachable and test_verify_fails_when_the_manager_cannot_be_queried (XDG_RUNTIME_DIR pointed at a bogus dir, DBUS_SESSION_BUS_ADDRESS absent from the subprocess env). Pre-fix: both fail. deploy/install-tools.sh is untouched (the call shape is unchanged).
• N1 — test_the_real_units_main_pid_is_our_daemon_and_this_process_is_not raced. The deployed shim is bash that execs python and start() returns on a non-zero MainPID; the test now calls _await_argv(main_pid, b"-m", <entry module>) before asserting. Exercised: 20 isolated reruns (one pytest process each) while the 64-test daemon modules ran concurrently with coverage: 20 passed, 0 failed.
• N2 — FAIL header now names only the tree(s) hit: the --dev-tree when a dev-tree hit exists, "a /home/*/workspace/ checkout" when a workspace hit exists, both joined by "and" otherwise. N3 — module docstring and the WORKSPACE_DIRECTIVES comment list the stop/reload/condition directives. N4 — two tests: test_both_reasons_on_two_lines_print_both_trailers_and_count_two (one dev-tree-only line plus one workspace-only line: header says 2 directive(s) and names both targets, both trailers print) and test_the_dev_tree_rule_matches_a_joined_continuation_line; plus test_a_workspace_only_header_does_not_name_the_dev_tree. N5 — the round-1 coverage note above is corrected: both missed pairs are OSError guards.
• Reviewer NOTEs NOT changed, with reasons. N6 Environment= indirection (a unit assigning a workspace path through Environment= and expanding it in ExecStart) is outside ADR-0067's scope: the workspace rule is defined on what ExecStart* / WorkingDirectory literally name. N7 a /root home: by ADR-0067 decision 4 the violation is /home/*/workspace/ literally, so /root/workspace is not one (now pinned by the M3 test).

Measured (fix round 2): 102 passed over test_check_prod_tree_separation.py + _paths + test_install_tools.py + test_ast_daemon_unit.py; 16 passed over test_retire_plain_units.py (two runs, 6 + 10); 64 passed over test_daemon_lifecycle.py

• test_daemon_coverage.py; 254 passed over tests/promotion/; 436 passed, 0 skipped in total. Coverage: scripts/check_prod_tree_separation.py 97 % (128 stmts, 4 missed: 175-176 and 224-225, the two OSError guards), refactor/codemod/daemon.py 92 % (274 stmts, 23 missed, all pre-P5 branches). Real-host gate still exits 1 with the same 4 directives (before-state, retired by the ship); its header now reads "4 directive(s) point production at a /home/*/workspace/ checkout" (no --dev-tree claim).

Process slip (fix round 2, recorded by the orchestrator): after the code commit the builder ran git checkout HEAD -- plans/ as hook-litter cleanup, which reverted its uncommitted plan edits for this record; they were re-applied and committed as 21a806cbd. No code was affected. The litter command names the six generated files explicitly for this reason — never a directory.

Review round 3 (2026-10-06, against a2e6b3913 / 21a806cbd): GO. All nine round-2 items confirmed closed; probes matched systemd for three/four trailing backslashes, blank line after a continuation, EOF mid-continuation, comment lines inside and outside a continuation; home seam evaluated at call time, gates.py unaffected; fixture end-to-end retire → verify → retire → verify exit codes 0/0/0/1; simulated post-retire host unit dir: gate exit 0 (18 units) with the default and the main-checkout --dev-tree. 436 passed / 0 skipped across five groups. Two NOTEs: this slip note was missing (fixed above); the ship-log row listed only the first commit (fixed at ship).

P6 — enable the primary instances and prove identity end to end

• R1 applied: systemctl --user enable --now orchestrata-{ast,codemod}-daemon@$(systemd-escape --path ~/workspace/orchestrata-ultimate) performed by install-tools.sh when ORCHESTRATA_PRIMARY_CHECKOUT is set (from ~/.config/orchestrata/.env? no — that file is secrets-only per ADR-0066; use env/host.env or a flag; ask in the brief, do not guess).
• Verify (also folded into orchestrata-provision skill step 0.2 "identity, not health"), for each enabled instance: tr '\0' ' ' < /proc/<MainPID>/cmdline | cut -d' ' -f1 is /opt/orchestrata-tools/.venv/bin/python; readlink /proc/<MainPID>/cwd is /opt/orchestrata-tools; identity deployed_commit == VERSION.
• Acceptance: a session opened in ~/workspace/orchestrata-ultimate gets a warm symbols answer (< 1 s, measured) and a codemod status from daemons whose code_root is /opt/orchestrata-tools/lib; systemctl --user --failed is empty of both daemons across a logout/login and a reboot; fs.inotify.max_user_watches measured against the instance count.
• P6b — promoter verify settles before it judges (from F-P6a-1, 2026-10-07). promotion/stages_verify.verify polls is-active + the kernel port and returns the moment every unit reads active and every port answers (8.6 s on the 1fed577b4 promote); a unit that runs for ~1 s, exits 1 and is restarted every RestartSec=10 is active for part of each cycle, so one poll can catch it there and the stage goes green (measured: the consumer exited 1 at +1 s on the ADR-0072 RuntimeError, the promote passed). Rule: once the poll succeeds, hold for max(RestartSec) + 1 s (read from the unit properties, not a constant) and then require ActiveState=active AND NRestarts == 0 for every restarted unit (NRestarts is already in identity_scope.UNIT_PROPERTIES). Proof test: a real transient unit with ExecStart=/bin/false, Restart=always, RestartSec=2 is refused by the verify; the same unit with /bin/sleep infinity passes. Ship's 5b (fixed in 325c721c6) stays as the independent second reading. Built 2026-10-07 (brief 4d726c78c + § 4 92e756596; build 7ac5e7f79; reviewer conditional GO, fix round 1 341678e58): stages_units.parse_systemd_timespan, stages_verify._unit_properties / _settle_hold(ctx, units, settle); 27 tests in tests/promotion/test_verify_settle_hold.py against real persisted user units (the incident shape sleep 0.8; exit 1, RestartSec=2 reads after the settle hold (3.0s): <unit> restarted 1 time(s); the in-hold crash reads not active: <unit> (activating); RestartSec=200 is refused in 0.1 s as exceeds the verify patience (10s)). Reviewer probe on the F-P6a-1 shape (sleep 1; exit 1, RestartSec=10): NRestarts rises in the same 0.2-s sample as the return to active, so the real verify refuses it in 11.0 s, every time. Bounded guarantee, stated in code and docs: a unit whose FIRST crash comes later than poll time + longest RestartSec + 1 s passes this stage (reviewer MAJOR-2: sleep 15; exit 1 with RestartSec=10 reached identity). Follow-up P6d — minimum-uptime floor (not built; sizes every promote's wait): require ActiveEnterTimestampMonotonic at least N s old at the re-read, N from the slowest known dependency timeout (the consumer's graph/Redis connect), to be measured before it is set.
• P6c — the graphs stage stops re-embedding the semantic graph on every promote (from F-P6a-2, 2026-10-07; design question, ADR-0074 Proposed, awaits Mohamed). Built only on Accepted; the brief is the ADR's Decision section.

P6a — the prod identity verifier judges tool-tree template instances against their own root (first item, 2026-10-07)

Measured on the S3 and PH1 ships (s3_ship2.log, ph1_ship2.log): check_identity(root=/opt/orchestrata, …) discovers orchestrata-ast-daemon@home-mohamed-workspace-…wt-SPEC-110.service through SERVICE_GLOBS and refuses on six lines: c0 "resolves outside ROOT and system prefixes" for the three %I-derived tokens (--repo-root /<repo>, --socket /<repo>/.claude/run/ast-daemon.sock, ExecStartPre=/usr/bin/test -e /<repo>/.git), /proc/<pid>/cwd expected /opt/orchestrata got /opt/orchestrata-tools, cmdline[0] expected under /opt/orchestrata got /opt/orchestrata-tools/.venv/bin/python, and start-time measured against /opt/orchestrata/VERSION. Every one of those judgements applies the product root to a unit ADR-0067 R1 defines otherwise: the instance parameter is DATA ("an enabled instance names data, not code") and the code root is /opt/orchestrata-tools (P3: the unit executes deployed code only). argv0 already passes because P3 put /opt/orchestrata-tools in SYSTEM_PREFIXES.

Executed design (no new decision; each rule is R1 or P3 applied):

1. Per-unit root. A discovered unit whose name carries an instance (@ before .service) AND whose ExecStart argv0 resolves under /opt/orchestrata-tools is judged against that root: root_resolved, venv_python (<root>/.venv/bin/python) and version (<root>/VERSION, same commit/deployed format, parsed with parse_version) are taken from the tools tree for checks c0, d/e (check_process) and the start-time floor. A template instance whose argv0 resolves elsewhere stays judged against the product root (a checkout-bound instance is still the defect).
2. Instance parameter is data. For such a unit, c0 tokens equal to the unescaped instance (systemd-escape --unescape --path <instance>, read from the unit name; the /%I form is an absolute path) or under it are exempt. Nothing else is exempt: a token outside the instance, the tools root and the system prefixes is still a mismatch.
3. Report names the root. Each such unit's UnitFacts carries root and the rendered report prints judged against /opt/orchestrata-tools (ADR-0067 R1: instance parameter is data) once per unit, so a reader sees which record each verdict used.
4. A tools deploy restarts live instances. Measured 2026-10-07: the wt-SPEC-110 instance started 02:24 EDT and /opt/orchestrata-tools/VERSION was re-stamped at 05:07 EDT by the PH1 ship, so under rule 1 the instance is stale code (start-time below the tools deployed), the identity defect the verifier exists for. deploy/install-tools.sh gains, after its verification section succeeds and VERSION is promoted, a restart of every ACTIVE orchestrata-ast-daemon@*.service and orchestrata-codemod-daemon@*.service instance in the user scope (systemctl --user restart), logged per instance; inactive instances are left alone (worktree instances are runtime-only, P4). The P4 hook already replaces a stale daemon at session start; this closes the window between a deploy and the next session. Refinement after the first build (695d41954, adc7dbe0d): the block as first written ran only on the default prefix and so had no behavioural test, because install-tools.sh has no systemctl seam and its tests deploy to a tmp prefix, where restarting the user's live instances would be wrong. Orchestrator's pick (reversible, mirrors retire-plain-units.sh): the block moves to deploy/restart-tool-instances.sh, which takes the unit globs and the systemctl command from env (TOOL_INSTANCE_GLOBS, SYSTEMCTL), so a test drives it with a recording systemctl stand-in script and transient units it creates itself, and install-tools.sh calls it on the default-prefix path only. The restart arm then has a positive test: an active instance is restarted, an inactive one is not, a restart failure refuses with the "VERSION promoted; … did not come back" message.
5. Tests: tests/promotion/test_identity_tool_tree_instances.py (new; real transient template-shaped units through _identity_fixtures._transient_unit with a tools-shaped root under tmp_path carrying VERSION and .venv/bin/python): a template instance whose argv0 is under the tools root passes with instance-parameter tokens present and is reported as judged against that root; the same unit with a token outside both roots still fails; a template instance whose argv0 is under a workspace checkout still fails against the product root; a plain (non-template) unit is unchanged; start-time is compared with the tools VERSION. tests/deploy/test_install_tools.py: the restart block restarts only active instances and only the two template names (dry-run/SYSTEMCTL seam as the script's other tests use). Coverage ≥ 90% on identity.py, identity_checks.py, identity_scope.py.
6. Acceptance: promote --to prod verify no longer lists the AST-daemon instance; the only remaining refusals are R16b (ADR-0072) and the backup oneshot until its promote.

P7 — docs

• docs/ship-and-deploy.md §3 (/opt/orchestrata-tools): add ast-daemon, codemod-daemon and both template units; plugins/orchestrata/systemd/README.md and deploy/systemd/README.md: the templates' new contract (replace the U-T2 header comment in the AST unit with a pointer to ADR-0067); docs/ast-toolkit.md: the hook's three branches; the SPEC-110 codemod docs: how the daemon is started now. plans/affected-tests-manifest.json: bind both template units, the hook, and _daemon_entry.py to their tests.

PH1 — prod unit health (the two failed timer-driven oneshots)

Named after the second ship (2026-10-06); diagnosed 2026-10-07 after the S3 ship's prod promote ran to verify and refused on, among other things, orchestrata-kg-retention.service and orchgraph-backup.service in failed.

• Retention — template defect, fixed here. deploy/systemd/orchestrata-kg-retention.service is the one unit that gates on orchestrata-kg-ready.py without declaring Environment=ORCHESTRATA_ENV_PROFILE=prod and EnvironmentFile=-/opt/orchestrata/env/prod.env (measured 2026-10-07 over the eleven templates: six gate on kg-ready, five of them declare the profile, retention does not). Since ADR-0066 /opt/orchestrata/.env is secrets-only, so the gate falls back to the TEST profile and waits 120 s on bolt://10.53.2.10:7687 (/var/lib/orchestrata/backups/orchgraph/backup.log and the unit journal show every run failing in ExecStartPre that way, "did not answer within 120s after 59-61 attempt(s)", while the prod cluster answered). The backup unit got the same two lines on 2026-10-05 with the comment that names this defect; retention was missed. Fix: the two lines, with the same comment, plus a test in tests/deploy/test_deploy_assets.py that every deploy/systemd/*.service whose ExecStartPre runs orchestrata-kg-ready.py declares both lines, so the class cannot recur. The installed copy under ~/.config/systemd/user is byte-identical to the template, so the next prod promote (which runs deploy/systemd/install.sh) carries it.
• Backup — already fixed on master, waiting on the promote. The deployed prod tree (287826a9) carries the pre-8b2038b92 scripts/backup_orchgraph.sh: its MAIN-container lookup is a plain incus list (no --all-projects), and its snapshot-pull step is the ls | grep -v pipeline that set -e kills silently. Measured 2026-10-07: run by hand from this shell the deployed script exits 1 at "no Incus container has address 10.53.1.123"; run by the timer at 03:00 it finds orchgraph-data-2, logs "snapshotting database 'orchkg'" and exits 1 with no message, the shape 8b2038b92's header describes. Master (4fadfaac1) has the fixed script; no code change in this packet. The unit clears when the next prod promote reaches a green verify, which needs P6's verifier item and Mohamed's ADR-0072 ruling.
• Done when: the retention template carries the two lines; the new test fails on the old template and passes on the new (shown in the builder's report); the deploy tests pass; shipped; after the next successful prod promote, systemctl --user --failed lists neither unit (verified in that promote's ship-log row, not here).

3b. Ship log

Prod-promotion state after P5 (2026-10-06). The two P5 blockers are gone: the plain units are retired and the test-env record is GREEN. Promotion now reaches the real prod deploy and stops at verify on three things P5 never owned: R16b (Mohamed's ruling, ADR-0072 Proposed), the verifier's treatment of tool-tree template instances (folded into P6 as its first item), and two failed timer-driven oneshots on the host (new packet PH1, "prod unit health": retention unit env + backup exit-1 diagnosis). Prod tree is at 287826a9 with the consumer failed (as before the ship). Nothing on the host was left in a worse state than before: units restarted by the promote are back on 287826a9.

Corrections from the second ship (2026-10-06). (1) Prod promotion IS coupled to this plan: the plain units fail the prod identity verify on their path directives whether or not they are running, so nothing promotes to prod until P5 removes them — that is before the conversation-consumer question (ADR-0066 R16b) is even reached. (2) Why the ad-hoc daemon keeps "going inactive": it exits cleanly after its 2 h idle timeout (code=exited, status=0 at 23:52:28 after a 2 h 00 m 12 s run) and the plain unit's Restart=on-failure never restarts a clean exit. Not the promoter's doing. P3's Restart=always is the fix for the template; the ad-hoc unit was restarted by hand at 01:5x on the standing request and is removed by P5. (3) The GREEN test-env record gate let this ship through where it had refused P1 minutes earlier; the record's scope rule is undocumented in this plan and is to be read from promote_env.py before P6 relies on it.

4. Order and ship cadence

P1 → P2 → P3 → P4 are one ship each (each independently green and useful; P3 without P2 would install a unit pointing at a binary that does not exist, so P3 ships only after P2 is deployed). P5 ships with P4 or right after. P6 follows P5. P7 rides with whichever packet last touches the file. Within a packet the AST half is built and green before the codemod half starts.

Every ship is bash deploy/ship.sh from the packet's worktree; ship's install-tools.sh arm is what deploys this, so no prod promotion is involved — this plan does not depend on the open conversation-consumer endpoint (ADR-0066 R16b).

5. Risks

• Memory: N instances × (≤1.7 GB AST + ≤1.9 GB codemod). Mitigation: MemoryMax, idle exit (AST), instances only for checkouts a session actually opens. The codemod daemon has no idle exit today; whether it needs one is measured in P6, not assumed.
• codemod_fast native build in the installer: a C build on the host path. The toolchain pin already exists; the verify step refuses the pure-Python fallback so a silent degrade cannot ship.
• inotify watch limits with several instances watching large trees: measure fs.inotify.max_user_watches against instance count in P6; fail the verify step loudly rather than let watchfiles degrade.
• Shared code, divergent checkouts: the deployed serialize_codebase_ast.py parses whatever checkout it is pointed at; a checkout on an older branch may carry source the newer tool parses differently. Acceptable (the tool is read-only and falls back), noted so nobody reads a mismatch as a bug.
• Hosts without the tools deployment get no daemon at all after P4 (R1): AST queries take the slow path and codemod.py refuses with a message naming the installer. That is the intended behaviour, and it is visible — not a silent regression — because both paths print why.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks