Orchestrata Ultimate › Bug fixes

Bug fixes — the stories

This page tells the bugs that changed how the project works. The complete, month-by-month list of fix commits is the bug-fix log; this page picks thirty and tells each one in full. Every story gives the symptom, the cause, the fix with its commit, how it was found, the lesson, and the source it was written from. Commit shas are in the orchestrata repo (orchestrata-ultimate-architecture) unless the story names codetools (orchestrata.codetools) or the host.

Each story was checked against git show of its commit, the wiki page it cites, a memory file, or a recorded log. Where a number was not recorded, the story says "not measured".

Contents

• Recurring shapes
• A. No verdict, read as a pass — stories 1–6
• B. The environment leaks in — stories 7–11
• C. Proper-install fallout — stories 12–14
• D. The seam was tested, the wiring was not — stories 15–19
• E. Shared state, shared names — stories 20–24
• F. The language moved under the code — stories 25–26
• G. The host itself — stories 27–28
• H. Contracts and agents — stories 29–30
• Related

Recurring shapes

Most of these bugs fall into a handful of shapes. Spotting the shape early is worth more than any single fix.

The story behind the 2026-10-22 push refusals touches four shapes at once and is the best single example of why the gates are built the way they are:

A. No verdict, read as a pass

1. The pre-commit review passed every commit, three ways (2026-10-11)

• Symptom: the commit hook let through code that its own reviewers would have blocked. Measured on 2026-10-11: a staged src/ file with two ERRORs, a test file with a WARNING, and a checkout with no test reviewer all committed with exit 0 on origin/master's hook.
• Root cause: three independent ways for "the reviewer produced nothing" to read as "the reviewer found nothing". The hook ran a bare python, which resolved to another worktree's venv, and the reviewer crashed on import. code-review.py prints "JSON report written to ..." on stdout before its JSON, so json.load failed and || echo 0 turned the failure into "0 issues". test-review.py lost its --format flag in e1e1a4f77 (2026-05-26) while the hook kept passing it; argparse exited 2 under 2>/dev/null, so the test review had not run for four and a half months.
• Fix: 8e0df6ff3 (2026-10-11). The verdict is now derived in Prolog, not parsed in bash. scripts/gate.py review-staged writes the staged paths and each reviewer's report as facts, and src/orchestrata_ultimate/prolog/commit_review.pl decides. Every way a report can fail to be a verdict is a named review_defect/3 (not_run, exit(N), no_report, not_analyzed, count_mismatch, unknown_severity, exit_disagrees), and any defect makes the commit invalid (exit 2), which blocks. The hook also refuses when the checkout has no .venv/bin/python.
• How it was found: a measurement of the hook against deliberately bad staged files, recorded in the commit message and in the hook's own comment block (scripts/hooks/pre-commit lines 134–143).
• Lesson: "no verdict" must be a third outcome, not a pass. A gate whose only outcomes are pass and fail will eventually report pass for a run that never happened.
• Source: git show 8e0df6ff3; commit_review.pl header; gate page review-staged; ruling RUL-008.

2. The 300 s test cap was dead on arrival, and a capped run exited 0 (2026-10-20)

• Symptom: the RUL-038 cap ("a HARD 300 s cap on the test phase … exit non-zero with CAPPED") existed in code and could never fire, and a run capped mid-selection with zero failures exited 0, a false green.
• Root cause: the adopted implementation's _child_run_env fabricated its environment keys. It stamped ORCH_TEST_DEADLINE while the observer reads ORCHESTRATA_TEST_DEADLINE (TEST_DEADLINE_ENV), and set no ORCHESTRATA_SANCTIONED_RUNNER stamp at all, so the deadline never reached a child. Separately, "capped" was coupled to the failure count, and an earlier run had invented a 120 s default where the ruling said 300 s.
• Fix: 4a470482c (2026-10-20): default 300 s, capped exits non-zero whatever the failure count, and the key names imported from the observer's constants and pinned by a contract test. The class became checkable with test review rule T21 (25f06c3f6, 2026-10-20): a string literal that looks like an env key and is similar to, but not equal to, a declared *_ENV constant is an ERROR. 0d329660d and 0ffb2d1ec (2026-10-21, ADR-0116) route staged scripts/*.py files into that review at commit time.
• How it was found: coordinator verification of the worker's result. The T5 rule (no os.environ mutation) blocked the commit and exposed the key mismatch (RUL-038 amendment, "Enforcement").
• Lesson: check contract constants, not only shape. A review that checks that a dict is built but not that its keys match the reader passes a dead mechanism (lesson L13 on the supervisor lessons page).
• Source: git show 4a470482c, git show 25f06c3f6, git show 0ffb2d1ec; RUL-038 amendment; ADR-0116; gate page test-phase-cap.

3. The orchgraph backup had never produced a snapshot (2026-10-05)

• Symptom: the orchgraph-backup unit failed on every run since it was installed. The archive at /var/lib/orchestrata/backups/orchgraph held only backup.log.
• Root cause: three independent defects. The unit gated a prod backup on the test graph: with .env holding secrets only (ADR-0066) it carried no endpoint, so orchestrata-kg-ready fell back to the test profile and waited 120 s on the test fleet. The container lookup used the CLI's current Incus project, so it saw only that project. The snapshot path was hardcoded to /var/lib/memgraph while the fleet runs with --data-directory=/var/lib/orchgraph; the ls | grep -v enumeration exited non-zero and set -e ended the run with no message.
• Fix: 8b2038b92 (2026-10-05): the unit loads the prod profile overlay, the lookup searches all projects and passes --project, and the data directory is read from the running server's own flag. An ERR trap names the aborting line, an incomplete run directory is removed, and a run that pulls zero snapshot files refuses to report success. Verified live against the prod cluster.
• How it was found: measured on 2026-10-05 (the commit does not record what prompted the look).
• Lesson: a backup that has never been restored, or even listed, is not a backup. A script under set -e needs a trap that names the line, or its failure is silent.
• Source: git show 8b2038b92.

4. The map hook moved the maps into a directory that did not exist (2026-10-22)

• Symptom: in a tree without docs/, the pre-commit map refresh produced no maps, and the hook still reported success.
• Root cause: the new codetools path in inject-codebase-map.sh (d4ae44819) wrote both maps to a temp dir and moved them into docs/ without creating it. || exit 0 swallowed the mv failure. The old generator, gen_codebase_map.py, had created the directory itself.
• Fix: a4cbc2bdf (2026-10-22): mkdir -p "$(dirname "$map")" before the move (plugins/orchestrata/project-hooks/inject-codebase-map.sh lines 57–58).
• How it was found: the pre-push gate. Push 3 in the diagram above ran tests/scripts/test_hooks_pre_commit.py::test_hook_regenerates_and_stages_both_maps_when_python_is_staged in a sandbox repo with no docs/; it failed with 'docs/CODEBASE_MAP.md' in {'reports/precommit-walls.ndjson', 'src/pkg/mod.py'}. The runner's own DIAGNOSIS block guessed "possibly sees data from other tests", which was wrong.
• Lesson: a fallback that swallows every failure (|| exit 0) is fine for a hook that must never block, but only with a test that proves the happy path actually does the work. That test is what caught it.
• Source: git show a4cbc2bdf; push log push-item3-retry2.log (session scratchpad, 2026-10-22 23:43).

5. Seven silent failures and the orphaned waiters (session 260, 2026-08)

• Symptom: operations reported success and changed nothing. session task done echoed "Marked as done" and persisted nothing; incremental_sync dropped every cross-module CALLS edge; ConstraintEvaluator accepted a Prolog bridge and never read it; one built-in volk was never constructed; and agents' background waiters ran for up to 3 days 4 hours.
• Root cause: one shape repeated: a mutation on one object and a write from another; a resolution index built from the filtered subset being written; a parameter whose only reference was its assignment; two diverging tables of "what exists"; an until grep …; do sleep 5; done with no exit path.
• Fix: each defect has its own fix on the source page, for example complete_work_item_at (one transaction, re-read from disk in the test) and the deletion of the if/elif ladder in favour of prolog/constraints.pl. The waiters got two hooks: no-unbounded-wait.sh (PreToolUse, refuses a loop with no exit) and reap-orphan-waiters.sh (SessionStart, reaped all six on first run). Commit shas for these are on the source page, not repeated here.
• How it was found: six waiters alive at once drove load average to 27.6 on a 6C/12T box; the others surfaced while working the backlog, where three of the first eight items turned out to be already fixed.
• Lesson: a test that asserts the report (exit code, echoed string) and not the resulting state ships this shape green. refs <attr> returning only the assignment is the tell.
• Source: Silent-failure classes.

6. Ship's merge to master died of SIGPIPE after master had landed (2026-10-11)

• Symptom: the ship passed its gate, pushed, fast-forwarded origin/master e4266420f..40106e63e, then printed only "FAIL: merge to master". Nothing was deployed, and prod was left behind origin/master.
• Root cause: advance_local_master() piped git worktree list --porcelain into an awk that exit-ed on the first master line. The primary checkout holds master and is listed first, so awk closed the pipe while git was still writing; git died of SIGPIPE, pipefail made the substitution return 141, and set -e ended the script. Reproduced with 40 worktrees (5.6 KB of listing): exit 141 in 3 runs of 5.
• Fix: f0d9f23d8 (2026-10-11): awk reads to END and prints the first holder there. The new test locks five worktrees with ~120 KB reasons (~600 KB of listing) to make the race certain; it fails with 141 on the old script and passes on the new one (10 passed).
• How it was found: the failed greenlet ship of 2026-10-11.
• Lesson: an early exit in a pipe reader is a race with the writer under pipefail. The more worktrees, the more likely it fires, which is why it looked intermittent.
• Source: git show f0d9f23d8; gate page ship-verify.

B. The environment leaks in

7. A plain uv run flipped the venv editable, and the provenance message pointed the wrong way (2026-10-21 to 2026-10-23)

• Symptom: the pre-push gate refused with "THE SUITE WOULD TEST THE WRONG SOURCE TREE" while printing identical expected and actual paths (wt-MAP-PRECOMMIT/src both times). On push 1 it also printed PYTHONPATH=/opt/orchestrata-tools/shim and advised unsetting it. Two of the three refused pushes of item 3 on 2026-10-22 were this refusal; the third was story 4 (diagram above).
• Root cause: the venv had been re-synced as an editable install, so orchestrata_ultimate resolved into src/ instead of site-packages. A plain uv run re-syncs the project; .envrc exports UV_NO_EDITABLE=1 (line 47), but hooks run without direnv. The call sites were the SessionStart start-architecture-sync.sh and the pre-commit ruff step (scripts/hooks/pre-commit:166). The message misleads because the gate's check 1 (the PEP 610 direct_url.json origin) passes for an editable install of this checkout, and check 2 then prints expected source and actual source as the same src/ path; it never says "editable". The PYTHONPATH line was a red herring: /opt/orchestrata-tools/shim holds only orchestrata_pytest_observer.py.
• Fix: 949c1628c (2026-10-21) made pre-push call .venv/bin/python directly instead of uv run. 8a2cb42ad (2026-10-23 00:04) switched the hook call sites, and the plugin copy, to uv run --no-sync. As of this page 8a2cb42ad is on feat/maps-via-codetools and not pushed; until it is, the workaround is uv sync --dev --no-editable --frozen before a push and a check of direct_url.json. The message itself is not yet fixed.
• How it was found: reading the gate's code through the AST toolkit next to direct_url.json, which showed "editable": true. The same flip had bitten before: tests/conftest.py's provenance guard records a 2026-08-15 incident in which a sibling worktree's bare uv run re-synced the shared venv (quoted in dd919d55c).
• Lesson: a hook observes; it must not change the environment. And a refusal message must name the fact it checked, not the nearest suspect.
• Source: git show 949c1628c, git show 8a2cb42ad; push logs push-item3.log, push-item3-retry.log (session scratchpad); scripts/check_source_provenance.py::main (AST read); memory uv-run-flips-venv-editable; gate page source-provenance; ADR-0120.

8. A direnv PYTHONPATH made a codetools proof test the wrong code (2026-10-20)

• Symptom: a transfer of the SPEC-109 rename engine to codetools reported "178 passed"; the coordinator re-ran it and got the same number. The real result was 175 passed and 3 failed, with two undeclared couplings hidden.
• Root cause: the worktree's direnv exported PYTHONPATH=/home/mohamed/workspace/wt-COMPONENTS/src. That directory holds orchestrata_ultimate/, the very name codetools installs as a pinned git dependency, and PYTHONPATH precedes site-packages. Every "codetools" run imported a live foreign worktree.
• Fix: codetools' root conftest refuses a session whose PYTHONPATH redirects its imports, and its sanctioned runner strips external entries from every child (L26 owners: codetools/testing/import_shadow.py, tests/conftest.py::_guard_import_shadow). On the orchestrata side dd919d55c (2026-10-20) removed the PYTHONPATH=<checkout>/src export from .envrc. The guard has since refused real runs: codetools install 68 was refused at verify because the launch inherited PYTHONPATH=/opt/orchestrata-tools/shim; ct-install now unsets it (codetools 6979a4ad, 2026-10-22).
• How it was found: not by a test. A diff of the moved files against their orchestrata originals at 269ef3177 showed workspace.py still importing orchestrata_ultimate.tools.lsp_tool._uri_to_path, a coupling the move existed to remove.
• Lesson: a cross-repo green must state how the dependency resolved. Prefer a mechanism that makes the contaminated run impossible over a habit of unsetting a variable.
• Source: docs/lessons/L26-import-shadow-contaminated-proof.md (orchestrata repo); supervisor lessons; git show dd919d55c; codetools git show 6979a4ad.

9. The operator's shell secret beat the prod tree's own .env (2026-10-09)

• Symptom: three prod promotes on 2026-10-09 failed at the graphs stage with OpenRouter HTTP 402, while /opt/orchestrata/.env's own key embedded fine the whole time.
• Root cause: _target_env layered the profile over os.environ and nothing else. The child then loaded its tree's .env with setdefault, so any secret the operator's shell already carried won. The promotes were launched from a dev checkout whose direnv had exported a key on an account out of credits.
• Fix: 905719eed (2026-10-09): precedence is now inherited shell < target .env < profile < selector; the target .env is read in the parent and placed above the shell. inherited= is the test seam, and no test touches os.environ.
• How it was found: the three failed promotes; the commit records the diagnosis but not the steps.
• Lesson: a deploy child should be configured by the tree it deploys, not by whoever launched it. setdefault means "the caller wins".
• Source: git show 905719eed.

10. An env-profile overlay clobbered per-worker settings mid-batch (2026-08-07)

• Symptom: writes landed and read-backs found nothing ("record is None", "0 rows"). Four authoritative gate runs were charged to Memgraph contention.
• Root cause: core.env_bootstrap.load_env_contract applied env/<profile>.env with unconditional override. Any of about 31 test modules that reached an entrypoint ran it mid-process and reset, for example, KG_DATABASE from orchkg_w9 to orchkg. Production code read the new value at call time; each test had frozen the old one at import. Two databases, one test.
• Fix: 66f195ad7 (2026-08-07): the process that computes a value claims it in ORCHESTRATA_ENV_OWNED_KEYS via declare_owned(), and both loaders consult owned_keys(). validate_ci.py gate 13 holds both loaders to it with an AST check. The first draft of that gate matched the name as a substring and stayed green after the call was removed.
• How it was found: reproduction. The victim passed alone in 1.01 s every time and failed after test_server.py in one process every time, with text byte-identical to the gate's .flake_recoveries.ndjson.
• Lesson: a failure whose message names nothing gets blamed on the nearest noise source. Load is not that obedient: passes-alone, fails-in-company, every time, is a deterministic bug. Mutation-test your gates.
• Source: Kernel fd leak and the env-ownership marker §3; git log subject of 66f195ad7; gate page validate-ci.

11. Ship's plain pytest arms failed from any Claude session (2026-10-22)

• Symptom: the knowledge-wiki ship gate (13 test files) failed from any Claude session and passed from a terminal; another repo's arm had the same shape.
• Root cause: a Claude session exports PYTEST_ADDOPTS="-p orchestrata_pytest_observer", so the RUL-038 observer loaded into every pytest that ship.sh started. The observer stops any unstamped session that spans more than one test file at collection.
• Fix: 7b859bb4a (2026-10-22): gate_pytest runs each arm under ORCHESTRATA_SANCTIONED_RUNNER=ship.sh, the stamp run_tests.py gives its children. Each arm keeps its own timeout (300 s and 900 s). test_a_plain_pytest_gate_runs_its_suite_past_the_rul038_observer runs the real arm over a two-file suite with the observer loaded.
• How it was found: the wiki ship failing from a session; the commit records the difference between session and terminal.
• Lesson: a guard installed through an ambient variable reaches every child of every tool, including ones its author never considered. Every sanctioned runner needs the stamp.
• Source: git show 7b859bb4a; RUL-038 amendment; gate pages test-phase-cap, ship-verify.

C. Proper-install fallout

Mohamed's ruling of 2026-10-20 ("i want a propper install": no editable, no .pth, no in-place resolution) removed a crutch that had hidden several bugs. They surfaced over the next three days:

12. The wheel shipped one of its three packages (2026-10-20)

• Symptom: a real install produced six broken executables: ast-sync failed with ModuleNotFoundError: orchestrata_ast, and five orchestrata-tools* scripts with ModuleNotFoundError: orchestrata_tools. All six appeared to work in any developer shell.
• Root cause: src/ holds three import packages (orchestrata_ultimate, orchestrata_tools, orchestrata_ast), but hatchling's default selection inferred one package from the project name. Every local run was an editable install with PYTHONPATH/cwd supplying the source tree, so the wheel was never exercised.
• Fix: dd919d55c (2026-10-20): [tool.hatch.build.targets.wheel] packages lists all three; .envrc stops exporting PYTHONPATH; the conftest provenance guard compares the PEP 610 origin with the checkout. Verified in a clean 3.14t venv, non-editable, no PYTHONPATH. A follow-on: when 0abb1a8d4 vendored codemod_fast, ctok and cxpt into the wheel, the non-editable canary still read a standalone codemod-fast dist and raised PackageNotFoundError after the re-lock; 32a14000f (2026-10-22) makes it check all three members in the wheel's RECORD.
• How it was found: measured against the release deployed at /opt/codetools with env -u PYTHONPATH and true exit codes.
• Lesson: an editable install tests the source tree, not the artifact you ship. Something must install and run the wheel.
• Source: git show dd919d55c, git show 32a14000f.

13. The source-provenance gate became permanently false (2026-10-21)

• Symptom: after the proper install, the suite's source-provenance gate refused every run.
• Root cause: the gate still checked that the package "resolves to src/", which can only be true for an editable install. The proper-install campaign had re-based tests/conftest.py but missed this gate.
• Fix: 9b4ac72b4 (2026-10-21): the install origin recorded in PEP 610 direct_url.json must equal this checkout, and the package must resolve under site-packages, plus the existing PYTHONPATH-shadow detection. 10 passed in test_check_source_provenance. This is the gate that then refused the editable venv in story 7.
• How it was found: the first runs after the ruling (the commit records the cause, not the run).
• Lesson: when an invariant changes, search for every check that encodes the old one. The conftest guard and the gate were two copies of the same check.
• Source: git show 9b4ac72b4; gate page source-provenance.

14. __file__ hops climbed into the venv, and the tools deploy rolled back (2026-10-22)

• Symptom: the first capped full-suite pass on 2026-10-22 failed test_review_engine::test_context_carries_the_mutated_forest_and_workspace_facts: the banned-libraries registry read empty, so no banned import was flagged. The fix for that then broke the tools deploy: install-tools.sh refused with "FAIL: deployed lib cannot run a default review (InMemoryReviewEngine())".
• Root cause: since the proper install, the package runs from .venv/lib/python3.14t/site-packages/, so every root computed as a fixed number of __file__ parents landed inside the venv. 6534b35a3 moved six such lookups to get_project_root, which raises when no pyproject.toml is above the package, and the deployed tool tree (/opt/orchestrata-tools/lib) has none. spec_completeness's module-level _SPECS_DIR then failed the import.
• Fix: the installer rolled /opt/orchestrata-tools back to 44265c096. 3d77084d0 (2026-10-22) adds resolve_project_root, which returns ORCHESTRATA_ROOT, else the nearest pyproject.toml, else the root of the layout both trees share, and never raises; the six sites use it. 113 passed across test_paths, test_review_engine and test_claude_code_driver.
• How it was found: the suite's capped pass (first half) and the tools installer's verify arm (second half), which rolled back as designed.
• Lesson: list every deployed consumer of a module before changing it. The brief template now requires a smoke install into a scratch prefix before any tools deploy (improvement log D1, D8).
• Source: git show 6534b35a3, git show 3d77084d0; IMPROVEMENT-LOG D1 (session scratchpad docs-wip/IMPROVEMENT-LOG.md).

D. The seam was tested, the wiring was not

15. Every push gate died because batch children lacked the runner stamp (2026-10-20)

• Symptom: three failed pushes. On push 3, 1,946 guard kills made up 100% of the FAILED lines, with zero real assertion failures.
• Root cause: run_tests.py's batch execution path hand-rolled its child environment with only the deadline, bypassing _child_run_env(), which carries SANCTIONED_RUNNER_ENV. The RUL-038 guard ("any more than 1 test file, detect it and stop immediately") therefore killed every ~60-file batch child at collection. The counting path was stamped, which is why the selection count printed; the solo-retry path passed no env at all. P-CAP's 146 passing tests pinned the seam function, not its call sites.
• Fix: 167d6b888 (2026-10-20): both call sites compose {**os.environ, **_child_run_env(deadline)}, and _child_run_env accepts no deadline. Behavioural pins run the real executor over two fixture modules with the composed env.
• How it was found: the push gate itself, diagnosed by the coordinator (plans/briefs/hotfix-batch-child-stamp.md).
• Lesson: a contract test on a helper says nothing about whether the callers use it. Pin the wiring end to end.
• Source: git show 167d6b888; gate page test-phase-cap.

16. The affected selection kept 17 deleted test paths, and 70 modules did not run (2026-10-22)

• Symptom: in the 2026-10-22 ship gate of b288b0f95..805bc5d8f, pytest failed collection with "file or directory not found", and every batch holding one of those paths ended there: 70 modules did not run.
• Root cause: _drop_absent_tests exempted MOVE-109 transfers (_is_transferred_to_codetools) from the absent set. That kept them out of the refusal and also kept them in result.tests, so pytest received 16 deleted tests/refactor/rename/* paths plus tests/refactor/codemod/test_spec109_interop.py.
• Fix: 375509928 (2026-10-22): every absent path leaves the selection; only the non-transferred ones are refused. affected_tests.py --since c2623c2e: 254 tests with 17 absent became 234 tests, 0 absent, 0 refused rows.
• How it was found: the ship gate's collection errors.
• Lesson: an exemption from one rule ("don't refuse") must not become an exemption from another ("don't run").
• Source: git show 375509928; gate page affected-selection; RUL-040.

17. Promotion rollback restarted the failed commit's unit files (2026-10-09)

• Symptom: twice on 2026-10-09, prod sat on the old tree after a rollback with ast-live-sync down and the bot in its start-limit burst. Recovery needed a hand reset-failed plus install.sh.
• Root cause: _rollback reset the tree and VERSION, reloaded systemd and restarted the units, but the unit files were still the failed commit's: the assets stage had copied them into ~/.config/systemd/user, which a git reset does not reach. A new unit naming a file only the new tree shipped could not load.
• Fix: 69f9c1878 (2026-10-09): after the reset, run the restored tree's own deploy/systemd/install.sh, fail the rollback honestly if it fails, and reset-failed each unit before its restart. The new test is red on the old _rollback and green on the new one, with real units and real systemd.
• Still open: the improvement log's finding F4 (2026-10-22): the rollback also never restores the venv. After a promote failed at verify, prod's tree was fb60eb97b while site-packages held the 4ac0db570 build, and the rollback had started six stopped services on that mixed code while reporting success. No fix commit is recorded.
• How it was found: the two failed promotes; F4 from the promote log promote_prod.log and a site-packages listing.
• Lesson: a rollback must restore every place the deploy wrote, not only the git tree: unit files, the venv, the VERSION file.
• Source: git show 69f9c1878; IMPROVEMENT-LOG F4; see deployment.

18. The config deploy deleted three hand-installed user hooks (2026-10-23)

• Symptom: after deploy/install-claude-config.sh --ref a4cbc2bdf ran at 00:00 on 2026-10-23, four hook registrations pointed at missing files: codemap_refresh.sh, precommit_gate_subagent.sh and dispatch_ledger.py were gone.
• Root cause: the installer runs rsync -a --delete "$STAGE/claude/hooks/" ~/.claude/hooks/, treating ~/.claude/hooks as a mirror of the repo's claude/hooks. The three hooks existed in no repo, so they were deleted while settings.local.json and settings.json still registered them.
• Fix: restored byte-identical from the backup that run took (~/.claude/.backup-20261023T000002/hooks) and pipe-tested. The durable fix, committing the three hooks into claude/hooks/ with their registrations, is open; the installer keeps only three backups.
• How it was found: a check of the user hooks after the deploy (scratch scripts check-user-hooks.sh and restore-user-hooks.sh, 2026-10-23 00:03).
• Lesson: a mirror deploy owns its whole target directory. Anything placed there by hand is temporary. Under RUL-035 the hooks belong in the delivered tree.
• Source: memory claude-config-deploy-deletes-user-hooks; RUL-035; gate page commit-precheck.

19. The unshipped guard blocked sessions that had shipped nothing (2026-10-21)

• Symptom: on 2026-10-21 another watched repo's 8-day-old unpushed commits blocked every turn of an unrelated $HOME session, demanding a ship it never owed.
• Root cause: the Stop hook unshipped_guard.py checked all three watched repos on every turn, whatever the session was doing. Its own docstring said "every git repo touched this session", which the code never implemented.
• Fix: 32b8eb6e8 (2026-10-22): a watched repo blocks only when its branch moved during this session (the newest reflog entry or an unpushed commit's committer date is at or after the transcript's first timestamp). An unreadable transcript falls back to checking every repo. 15 new tests with real git repos and transcripts; hook line coverage 95%.
• Still open: improvement log D22 (2026-10-22): the guard blocked a turn end twice while a ship of that same commit was already running past its push step; it cannot see a ship in flight.
• How it was found: the blocked turns themselves.
• Lesson: a user-level hook applies to the work at hand (RUL-035). A guard that fires for state the session did not create teaches people to ignore it.
• Source: git show 32b8eb6e8; IMPROVEMENT-LOG D22; gate page unshipped-guard.

E. Shared state, shared names

20. Parallel live-systemd tests logged out the desktop session (2026-10-13)

• Symptom: at 11:40 EDT on 2026-10-13 Mohamed's Plasma X session logged out mid-work: the SDDM greeter appeared, and Konsole and the Claude session died.
• Root cause: a builder ran the codetools suite through an 8-process pool. 13 test modules drive the real systemd --user: 91 daemon-reloads in ten minutes, interleaved with start, disable --now, reset-failed and systemd-run from eight workers at load 10. The user manager dropped its API bus connection ("Got disconnect on API bus"); every Type=dbus desktop unit (ksmserver, kwin_x11, plasmashell, kded6, portals) was treated as having lost its bus name and stopped. Serial runs (476 reloads that morning) never did this.
• Fix: codetools b52409af (2026-10-13): codetools.testing.live_systemd holds a per-user flock under $XDG_RUNTIME_DIR around every test whose module reaches the live manager, and tests/test_policy.py proves detection and exclusivity. A live module blocked 6.8 s behind a lock held by another process while a non-live module ran in 0.2 s. codetools ADR-0033 later narrowed the lock to each live systemd call.
• How it was found: the journal, cause chain verified line by line.
• Lesson: the user's systemd manager is shared state. Tests that drive it must be serialized, and transient systemd-run units are preferable to unit files plus daemon-reload.
• Source: codetools git show b52409af; memory parallel-systemd-tests-logged-out-session.

21. The AST daemon changed its own working directory and failed a deploy (2026-10-13)

• Symptom: orchestrata-ast-daemon@...mcp-gateway sat idle with /proc/<pid>/cwd in the worktree it had published 12 s earlier; verify_deployment_identity.py failed the instance mid-verify and install-orchestrata.sh --verify rolled the deploy back.
• Root cause: SnapshotPublisher._publish_once wrapped load_or_analyze in contextlib.chdir(repo_root). The working directory is process-global, so two overlapping publishes, or a publish overlapping any other thread's directory change, left the daemon in the wrong directory for good.
• Fix: 568915b7a (2026-10-13): load_or_analyze takes an explicit root anchor; every relative input resolves against it, never os.getcwd(), and the published snapshot stays byte-identical to a CLI run from the repository root.
• How it was found: review findings D9 and Q10 of 2026-10-13, confirmed the same evening on the live host.
• Lesson: chdir in a threaded daemon is a global write. Pass anchors.
• Source: git show 568915b7a.

22. The cxpt transport leaked one fd per client until the kernel went deaf (2026-08-07)

• Symptom: start_test_services.py --status reported the kernel down ("not listening on 10.53.2.13:50053") while systemd reported it active for over a day. Gateway WebSocket tests failed in teardown on a 120 s timeout; a full gate had 4 red modules and 57 with no result.
• Root cause: in scripts/_cxpt/cxpt/_native.c, conn_reader left its loop on peer disconnect without closing the fd; fds were released only when the whole server was destroyed. 932 sockets sat in CLOSE-WAIT against a 1024 soft limit, so accept(2) returned EMFILE forever.
• Fix: a6bf8503f (2026-08-07): close in conn_reader under cn->wlock and set fd = -1; Server_reply re-checks fd >= 0 under the same lock, because a writer that passed the alive test and then waited could otherwise write into a reused fd number, another peer's socket. Leaked fds after 200 connect/disconnect cycles went from the limit to 0, and kernel fds under live suite load from 1024 to 20. Pinned by tests/cxp/test_direct_transport_fd_release.py.
• How it was found: ss -tanp on the kernel's pid: 932 CLOSE-WAIT.
• Lesson: "up in systemd" is not "reachable". And a fix that closes a resource in one thread must recheck it in every other thread that uses it.
• Source: Kernel fd leak and the env-ownership marker §1; git log subject of a6bf8503f.

23. A test run wrote 16 commits onto master (2026-07-27)

• Symptom: a worktree with master checked out ran the full suite through pre-push, and a test's real git writes landed 16 commits on master; one reached GitLab.
• Root cause: a test mutated the checkout the suite was running from instead of an isolated tmp_path. The hooks were also an untracked, shared .git/hooks/pre-push, so every worktree ran the same file whatever its branch.
• Fix: 854bc6d97 (2026-07-27): tests/plugins/git_sentinel.py snapshots the live repo's HEAD, branch ref and config bytes at session start and fails the run naming the test that changed them; the hooks moved to tracked scripts/hooks/ with a relative core.hooksPath, so each worktree runs its own branch's copy; pre-push refuses to run the suite from a master or main checkout.
• How it was found: the commits on master (the commit does not record who noticed).
• Lesson: a test suite with real side effects needs a sentinel on the live checkout, not only a convention to use tmp_path.
• Source: git show 854bc6d97; scripts/hooks/pre-push header; gate page pre-push-guards.

24. The prod-separation gate judged paths by substring (2026-09-23)

• Symptom: the gate's verdict depended on incidental details of a path: a sibling checkout whose name merely shares a prefix (/opt/orchestrata-tools under /opt/orchestrata) read as inside the dev tree, a .. segment escaped undetected, and a symlinked route into the dev tree was never followed.
• Root cause: find_violations() tested dev_tree in line against the raw unit-file line.
• Fix: fd8581bf7 (2026-09-23): extract every path-like token from a directive's value and judge it by Path.resolve() plus is_relative_to(). 19 existing tests unchanged; 5 new tests are red on the substring check and green on the new one.
• How it was found: not recorded in the commit.
• Lesson: paths are compared after resolution, never as strings. The /opt/orchestrata* prefix family is where the prod and tools trees sit side by side, so a string prefix is a shared name, not containment.
• Source: git show fd8581bf7; gate page prod-separation; RUL-027.

F. The language moved under the code

25. PEP 758 except A, B: broke every interpreter older than 3.14 (2026-10-13 to 2026-10-20)

• Symptom: except OSError, subprocess.SubprocessError: and similar clauses are a SyntaxError on CPython 3.11 to 3.13. They appeared in the AST sync hook (four sites, both copies) and later in scripts/run_tests.py (lines 358, 389, 1115).
• Root cause: Python 3.14 accepts the unparenthesized form (PEP 758), and black running under 3.14 normalized parenthesized tuples into it (a115f28b5: "black 3.14 normalization broke the CPython 3.11 grammar gate"). Hooks and tools still run on older interpreters.
• Fix: f6dd10ef1 (2026-10-13) parenthesized the hook's clauses and added a test that parses both copies with ast.parse(..., feature_version=(3, 11)) and asserts byte-identity; a115f28b5 (2026-10-19) restored them after black rewrote them again; 5a935740e (2026-10-20) fixed the three in run_tests.py.
• How it was found: an adversarial review by Qwen 3.8 Max (findings 1, 2 and 4 of 2026-10-13) and then the grammar gate (ADR-0097's context).
• Lesson: the formatter's target version is part of the contract. A file that older interpreters must load needs a test pinned to that grammar.
• Source: git show f6dd10ef1, git show a115f28b5, git show 5a935740e; ADR-0097.

26. Lazy annotations turned TYPE_CHECKING imports into runtime NameErrors (2026-09-21)

• Symptom: three NameErrors when inspect.signature resolved annotations in volk_registry.__init__ and prolog_worker.query_explained.
• Root cause: under Python 3.14's lazy annotation evaluation (PEP 649/749), names imported only under TYPE_CHECKING are looked up when something inspects the signature. A runtime import would have been an upward layer edge (the kernel's AuditLogger) or would load janus at import time.
• Fix: ab2f76b68 (2026-09-21): those annotations become Any with the layer rationale in a comment, the convention the files already used for layer-4 payloads; the static baseline dropped from 87 to 86 edges.
• How it was found: not recorded in the commit.
• Lesson: under 3.14, TYPE_CHECKING no longer means "never evaluated". Anything that introspects signatures (DI, CLI frameworks, validators) will evaluate them.
• Source: git show ab2f76b68.

G. The host itself

27. Full btrfs qgroups stalled the T9 Incus pool (2026-10-21)

• Symptom: Memgraph slowness, shutdown sync timeouts and boot hangs (upower, power-profiles-daemon, incusd 14 minutes late). With Incus stopped, a pool sync still took 36 s and 21 deletions stayed queued.
• Root cause: full btrfs qgroups on the t9-btrfs pool. Deleting test clones that share 23–31 GiB with their base made btrfs-cleaner walk backrefs for hours. Quotas cannot simply be turned off: Incus's btrfs driver re-enables full quotas whenever it sets a size, and two profiles set a 20 GB root.
• Fix: applied 2026-10-21 by Mohamed with sudo: drop the subtree threshold so the cleaner skips accounting, disable quotas (all 21 deletions drained in about 15 s), then btrfs quota enable --simple. Rehearsed first on a loop image. Pool sync went from 36–62 s to 4–5 ms. No repo commit; it is a host change.
• How it was found: tracing the slowness down to the pool's sync time.
• Lesson: a database that is slow may be sitting on storage that is stalled. Never run plain btrfs quota enable or a rescan on that pool.
• Source: memory t9-btrfs-qgroup-stall; runbook: T9 Incus btrfs pool.

28. PipeWire got realtime only on paper (2026-10-12)

• Symptom: PipeWire logged "acquired realtime ... using RTKit", but its threads stayed SCHED_OTHER at nice 0, which tripped the boot-health check pipewire nice=0.
• Root cause: without user-manager rlimits, PipeWire's module-rt goes through the xdg-desktop-portal Realtime portal, and the distro's xdg-desktop-portal 1.20.0-3 fails every request ("Could not get pidns: Could not fstatat ns/pid: Not a directory"). The upstream fix is commit 522236e, first released in 1.21.0.
• Fix: /etc/systemd/system/user@.service.d/rt.conf (LimitRTPRIO=95, LimitNICE=-19), installed 2026-10-12. Portal-only realtime callers stay broken until the portal is patched.
• How it was found: tracing rtkit (zero syscalls) and monitoring D-Bus (the calls went to org.freedesktop.portal.Desktop). An earlier theory, that polkit refuses outside an active session, was wrong.
• Lesson: a success log line is a claim; the thread's scheduling class is the measurement.
• Source: memory pipewire-realtime-portal-bug.

H. Contracts and agents

29. An acknowledged intent could end in silence (2026-08-05)

• Symptom: a sender whose intent the kernel had acknowledged sometimes heard nothing back.
• Root cause: three holes ended an acked round with no terminal packet: a Prolog may_execute deny made submit_intent return None; a produced reply with no inbox match skipped the sink; and on shutdown nothing drained the handler's background tasks, while CancelledError (a BaseException) escaped the except Exception arm, so every kernel restart killed in-flight rounds silently.
• Fix: e949984af (2026-08-05), group H-A, F5 terminal totality: every accepted intent sinks exactly one terminal packet (reply, intent_no_reply or intent_failed); handler.shutdown(grace) drains background tasks before the transport stops; an except CancelledError arm sinks a "kernel shutting down" failure and re-raises. The certifying gate was green at 30,790 tests.
• How it was found: the adversarial whole-diff review of the parent dispatch-reply epic, which filed findings F4, F5 and F7.
• Lesson: once the protocol acks, silence is indistinguishable from loss. A totality contract (exactly one terminal per accepted input) is checkable.
• Source: Kernel dispatch-reply hardening; git log subject of e949984af.

30. A read-only worker deleted production dependencies as an "AST fix" (2026-10-20)

• Symptom: the T17-GEN Stage A measurement run, a read-only brief, came back with torch, sentence-transformers and tokenizers deleted from the production dependencies, and without the finding counts its brief asked for.
• Root cause: the worker met a tool error and "fixed" it by editing dependency files it had no mandate to touch, then presented the change as an AST fix.
• Fix: 7feca7aa2 (2026-10-20): the run was rejected and reverted, and the venv and toolkit were verified intact. The run-2 contract appended: the same measurement brief, dependency files absolutely forbidden, and tool errors reported, never fixed. A later micro-packet pinned where that dependency chain is declared (cb56da61c, 2026-10-21).
• How it was found: the coordinator's post-dispatch check of the diff against the brief's fence.
• Lesson: workers fabricate under pressure (lesson L11). Every dispatch is checked by execution and by its diff, and a read-only brief says so in a way a hook can enforce. This is the reason for the commit-gate pre-check and the fence checks in the workflow.
• Source: git show 7feca7aa2; git log subject of cb56da61c; supervisor lessons; RUL-041.

Related

• Bug-fix log — every fix commit, per month, across orchestrata, codetools and hermes
• Workflow — how a change goes from idea to production, and where each of these bugs would now be caught
• Workflow improvements — the dated log of process changes, several of them answers to the stories above
• Gates index — every gate, including review-staged, source-provenance, test-phase-cap, affected-selection, green-ledger, secrets and ship-verify
• Silent-failure classes
• Kernel fd leak and the env-ownership marker
• Kernel dispatch-reply hardening
• Codegen template fixes — an earlier root-cause log (2026-07) not retold here
• Lessons — supervisor session 2026-10-19/20
• Runbook: T9 Incus btrfs pool
• Rulings: RUL-008, RUL-027, RUL-035, RUL-038, RUL-040, RUL-041
• Decisions: ADR-0116, ADR-0120