Orchestrata Ultimate › Plans › spec-110-merge-plan

SPEC-110 (codemod engine) — merge plan

Source: plans/spec-110-merge-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-27 · 421 lines.

Prose pending.

Plan text

SPEC-110 (codemod engine) — merge plan

Branch: feat/codemod-engine · Worktree: ~/workspace/wt-QWEN · Author: opencode Status when written: spec/design/briefs authored; implementation not started. Status 2026-07-27 (full read of the branch): implementation is well past scaffold — 5,607 LOC across refactor/codemod/** (24 modules incl. daemon/server/client/watcher, observability, persistence) and 7,204 LOC across 16 test modules. Only two NotImplementedError remain, both in passes.py. The W0 completion report claiming "skeleton code has no implementation" is stale. Rule for this plan: nobody but opencode writes in wt-QWEN until it reports done.

Intent (Mohamed, 2026-07-27): SPEC-110 lands, and it replaces existing modules. The overlap with the AST toolkit and the review gates is deliberate, not waste — see §7.

0. Do these FOUR things before writing implementation code

Each is minutes now and hours later.

1. Rebase feat/codemod-engine onto master. The branch sits on bfd0073a, which predates the graph-rename wave. Implementing on that base means writing against symbols that no longer exist — create_neo4j, Neo4jClient, Neo4jConfig, NEO4J_URI — and the engine is unwired, so import errors will not surface in the suite. Post-wave names are create_graph/create_async_graph, GraphClient/GraphClientConfig, BoltSyncRepository/BoltAsyncRepository, and the env seam is KG_ROUTING_URI (falling back to KG_URI) via core.graph_factory.resolve_uri — no literals.
2. Settle the REQ-id collision. spec.md numbers REQ-110-001..035 + 101..108; the generated checklists/requirements.md renumbers the same content 001..132, so the same id means two different things (its REQ-110-101 is a ComplexityAnalyzer, the spec's is a no-mocks NFR). The .feature files tag the spec's scheme, constitutional.md endorses the checklist's. Pick one — spec-authoritative per RUL-013 — before any @traces(reqs=[...]) is written, or every trace and the whole RTM join is wrong. Update: too late to prevent — the code now cites BOTH schemes. review.py uses the spec scheme (REQ-110-012 = ReviewPass ABC); txn.py cites REQ-110-050..057 and observability/* cites REQ-110-085..090, which exist only in the checklist. Against spec.md those ~20 ids resolve to nothing, so check-traces will call them ghosts and the RTM join drops those rows. This is now a rewrite of existing @traces, not a prevention — and it damages exactly the traceability the engine is meant to improve. Do it before the merge.
3. Regenerate the binding contracts in plans/codemod-engine-design.md. Its §3.2 signatures and §3.3 decisions describe the pre-expansion engine and omit the entire new surface (code-intelligence graph, persistence repository, events/observers, in-memory review engine, analysis passes). An implementer following §3.2 today builds the wrong object model.
4. Commit the work. As of 2026-07-27 the entire SPEC-110 implementation — 5,607 LOC of source and 7,204 of tests — exists only as untracked files in wt-QWEN. There is no git safety net: lose the worktree, lose the work. Nothing else in this plan matters if that happens. Committing costs a minute and does not interrupt opencode.

1. What the branch will contain

Mostly additive, but not purely — canonical_schema.py is edited, and pyproject.toml carries destructive changes that must not merge as-is (§4).

Scenario-count drift: spec §17 DoD says "12 feature files with ~81 scenarios", the implementation summary says 126, the tree holds 183. The DoD is unverifiable until one number is made true; fix the DoD, not the tests.

2. Base drift — the one real risk

The branch is based on bfd0073a, which predates the graph-rename wave now on master. It has NOT been rebased since. Two consequences:

1. pyproject.toml will conflict. master renamed the pytest markers (requires_neo4j→requires_graph, dropped the legacy neo4j marker, no_neo4j_cleanup→ no_graph_cleanup) and added the vendored grpcio-tools FT wheel pin; the branch predates all of it and adds basedpyright. Resolution: union. Take master's marker renames AND the branch's additions; never resolve by taking one side wholesale. Correction (measured 2026-07-27): this plan previously claimed opencode added bdd_features_base_dir + ~130 REQ_110_*/FR_* markers. Both are false — the branch has no bdd_features_base_dir and zero REQ_110_* markers (grep -c = 0). The pyproject conflict is therefore smaller than assumed; the feature files are bound by hardcoded absolute path instead, which is worse — see §9.4. Measured conflict surface: exactly two files, pyproject.toml and uv.lock. All 45 other committed paths are pure additions (git merge-tree against the merge-base).
2. Any codemod code touching the graph seam must use post-wave names (create_graph, GraphClient, BoltSyncRepository, KG_ROUTING_URI/KG_URI). If the branch was written against pre-wave names it will import symbols that no longer exist — check before merging, do not assume the suite will catch it (the engine is unwired, so import errors can hide).

Preferred sequence: opencode rebases onto master itself, before reporting done. It has the context to fix its own conflicts. Only if that is impractical do I merge and resolve.

3. Pre-merge checks (run in wt-QWEN, read-only until it reports done)

cd ~/workspace/wt-QWEN && export PYTHONPATH=$(pwd)/src
git log --oneline master..HEAD          # every commit authored by Mohamed, no AI attribution
git status --porcelain                  # must be clean
grep -rn 'neo4j' src/orchestrata_ultimate/refactor/codemod/ \
  | grep -v -e 'neo4j://' -e 'import neo4j' -e 'dialect'      # must be empty (plan §2 policy)
.venv/bin/ruff check src/orchestrata_ultimate/refactor/codemod tests/refactor/codemod scripts/codemod.py
.venv/bin/python scripts/serialize_codebase_ast.py standards
.venv/bin/python scripts/serialize_codebase_ast.py check-deps --include-tests
.venv/bin/python scripts/test-review.py tests/refactor/codemod/ --stdout
.venv/bin/python scripts/pyright_ratchet.py --max 283

Then the branch's own tests, parallel per module in ONE call, observer attached:

for m in tests/refactor/codemod/test_*.py; do
  .venv/bin/python -m pytest "$m" -q --no-header \
    -p orchestrata_ultimate.core.threading.pytest_live_observer > "/tmp/s110-$(basename $m .py).log" 2>&1 &
done; wait; grep -H -e passed -e failed /tmp/s110-*.log

4. Dependency gate — do this before the merge, not after

libcst, xxhash and pytest-bdd are new. On Python 3.14t a C/Rust extension that does not declare free-threading support re-enables the GIL process-wide, which fails the conftest GIL probe and reddens the entire suite, not just SPEC-110. Verify explicitly:

.venv/bin/python -c "import libcst, xxhash, sys; print('GIL enabled:', sys._is_gil_enabled())"

GIL enabled: False is required. If any of them flips it, the merge waits on a vendored free-threaded wheel (precedent: scripts/build_ft_grpcio.py, build_ft_tree_sitter.py) — do not merge a branch that turns the GIL back on.

This risk is no longer hypothetical. The branch's pyproject.toml comments out, under the banner # TEMPORARILY DISABLED:

• grpcio = { path = "vendor/wheels/grpcio-1.82.1-cp314-cp314t-linux_x86_64.whl" } — "already installed in venv, wheel build pending"
• all five tree-sitter* deps and their five vendored cp314t wheel pins

The grpcio pin is load-bearing: stock grpcio re-enables the GIL, which is why the FT wheel was built and pinned in the first place (see project-ft-prolog-blocked-by-tabling — "nor stock grpcio"). A local convenience in one worktree becomes a repo-wide GIL regression the moment it merges, and nothing gates a commented-out [tool.uv.sources] line — no test imports it, no gate reads it. Restore all six pins before merge; treat this as the highest-priority blocker because it is the only finding that escapes the branch and breaks master.

It also adds a new local path dep — codemod-fast = { path = "scripts/_codemod_fast" } with no-build-isolation-package. Confirm it declares Py_mod_gil = Py_MOD_GIL_NOT_USED (REQ-110-129) and that a fresh clone can build it, since the .so is not tracked.

5. Merge sequence

1. Blockers first: the SPEC-110 review findings that must be settled before code lands — the two colliding REQ-110-* numbering schemes (spec vs checklists/requirements.md), and the stale binding contracts in plans/codemod-engine-design.md. Both are cheap while opencode is still writing and expensive after.
2. Rebase feat/codemod-engine onto master (opencode), resolving pyproject.toml as a union.
3. Re-run §3 + §4 on the rebased branch.
4. Adversarial review of the completed spec + implementation (the earlier review was against a mid-write document and is parked as provisional notes only — re-run from scratch).
5. Merge --no-ff into master with a message naming SPEC-110 and the new deps.
6. One full suite from a worktree that does not have master checked out (see §6), then push.

6. Two hazards this session proved real — do not repeat them

• Never run the suite from a worktree with master checked out. On 2026-07-27 a pytest run drove git against the live repo: flipped .git/config to bare=true, wrote core.fsmonitor at a temp script, replaced the commit identity, committed fixture commits onto master, and pushed one to GitLab. Until the isolation guard lands, run the suite only from a feature-branch checkout, and diff .git/config + git rev-parse master before and after.
• The checkpoint hook commits onto whatever branch the main repo has checked out, including files written in other worktrees. It put SPEC-110 briefs, a .pl rulebase, a 16.5K-line session transcript and ~145 lines of SPEC-110 pyproject.toml config onto feat/graph-rename, and once committed a file mid-merge with conflict markers still in it. Before any merge to master, list added paths and strip anything that is not the branch's own work: git diff master..HEAD --name-status --diff-filter=A

7. Integration — what SPEC-110 replaces

The engine is not an addition alongside the AST toolkit and the review gates; it supersedes much of both. The value is not new facts — the AST toolkit and the orchkg KG already hold most of them — it is round-trips: warm, local, lossless, and writable, which none of the existing layers are (the AST toolkit is not lossless; the KG is out-of-process over Bolt to 10.92.53.160).

Governing rule: nothing is deleted without a proven parity case. The mapping below is read off the spec, not measured against behaviour. Several of these tools gate the suite; a gate is not something to swap on inference. Each retirement needs a side-by-side run on the real tree, same inputs, diffed outputs, before the old path goes.

Not replaced: refactor/rename/ (SPEC-109) — consumed, not superseded; REQ-110-020 forbids a second resolver. The AST toolkit's generators (gen-dto, gen-serde, gen-protocol, gen-skeleton, pattern) and check-traces have no SPEC-110 counterpart and stay.

The CODEBASE_MAP retirement is the largest single win. That @import costs ~31K tokens in every session and every subagent; CODEBASE_MAP_FULL.md is 263K tokens that killed four subagents in one night. Both exist only because there is no query seam cheap enough to call mid-task. A daemon answering "signature of X, its callers, its tests" in <100ms removes the reason for both. It is also the most reversible retirement — the file regenerates.

Retire in waves, ordered by blast radius, not convenience:

1. CODEBASE_MAP + ast_daemon/ast_api — high value, reversible, gates nothing
2. read-op parity → retire the overlapping serialize_codebase_ast.py subcommands
3. autofix / affected-tests / graph-sync
4. the review gates last, one at a time, each with a full green suite behind it

The wiring is where the payoff lives

Design decision D6 defers registry and volk wiring — refactor/__init__.py and agents/tool_registry.py are explicitly out of scope, and REQ-110-105 repeats it. Under RUL-023 that ships a CLI and a daemon nothing calls: built, not delivered. Every retirement in the table above depends on call sites pointing at the engine. Pull D6's wiring into the landing plan — otherwise we merge the engine and keep paying for the tools it was meant to replace.

Spec surface that is not built yet

Three FRs have no implementation to merge, and ~55 BDD scenarios with no target:

• prolog/codemod_review.pl does not exist. PrologSemanticReview is wired and every failure path returns [] under ERR-110-015 "graceful degradation" — so the safety gate always passes and no test can distinguish "checks passed" from "checks never ran". Either build the rulebase or make its absence a loud failure; a silently-passing safety gate is worse than none.
• InMemoryReviewEngine does not exist (FR-012, the 40 checks). review.py has 4 review passes. in_memory_review.feature carries 41 scenarios for it. This is also the direct blocker for retiring the review gates in wave 4.
• analysis/ was never created (FR-011's 7 analyzer classes). test_analysis_passes.py tests CIG query methods instead — reasonable coverage, different object model than specified.

9. Measured state — full investigation, 2026-07-27

Everything below is measured against the live tree, not inferred. Commands are read-only; wt-QWEN was not modified.

9.0 THE TRAP THAT VOIDS VERIFICATION — read this first

PYTHONPATH=/home/mohamed/workspace/orchestrata-ultimate-architecture/src is exported in this environment. It is not in ~/.bashrc, ~/.profile, ~/.zshrc, fish config, or ~/.env — it comes from the session environment itself, so it is present in every shell, and it cannot be fixed at the profile level.

Effect: it lands at sys.path[1] and outranks a worktree's own editable path at sys.path[6]. Inside wt-QWEN, import orchestrata_ultimate resolves to the main checkout — a different branch's source. Verified:

as-invoked      : /home/mohamed/workspace/orchestrata-ultimate-architecture/src/...  ← WRONG
env -u PYTHONPATH: /home/mohamed/workspace/wt-QWEN/src/...                           ← right

This voided my own first gate run in this session: standards and check-deps appeared to crash with ModuleNotFoundError: orchestrata_ultimate.core.neo4j_factory — the branch's script running against graph-rename's renamed source. Both gates pass clean when run correctly. It is the same failure that produced 30 phantom failures in a full suite run earlier.

Every command in this plan must be run with env -u PYTHONPATH. Two consequences for landing:

• Any test result opencode has produced in wt-QWEN is only valid if its shell lacked the same export. Confirm before trusting a green report.
• scripts/check_source_provenance.py (commit c3f0680c) is the gate for exactly this and is on master only. feat/graph-rename is 32 commits behind master and does not have it; neither does feat/codemod-engine. Whichever branch runs the post-merge suite must have it.

9.1 Topology — three unmerged branches

feat/codemod-engine is 328 commits behind feat/graph-rename. origin/feat/graph-rename is a70ec8d1 — the local branch is ahead of its remote.

Merge-order decision, and it is not free: if feat/graph-rename lands first, the codemod rebase spans 328 commits of vocabulary rename. If codemod lands first, it rebases across 175. The codemod source does not reference any pre-rename graph symbol (create_neo4j, Neo4jClient, Neo4jConfig, NEO4J_URI, neo4j_factory — grep returns empty across refactor/codemod/ and scripts/codemod.py), so the rebase is mechanical either way. Land codemod first, off master.

9.2 Free-threading — clean, better than the spec feared

Measured in wt-QWEN/.venv (Python 3.14.6t):

GIL enabled (baseline): False
  after import grpc   : False
  after import libcst : False
  after import xxhash : False

• REQ-110-107's LIBCST_PARSER_TYPE=pure fallback is not needed — libcst 1.8.6 is FT-clean.
• xxhash is FT-clean.
• The installed grpcio keeps the GIL off, so the venv is fine. But the pin is commented out in the manifest — a fresh uv sync pulls stock grpcio from PyPI and re-enables the GIL process-wide. It works here and breaks everywhere else. That is the whole risk of §4, stated precisely.
• codemod_fast is NOT installed (ModuleNotFoundError) despite being declared. Everything currently runs on _fast_fallback.py. The C/Python parity tests (REQ-110-129, Wave 5) therefore cannot have passed — there is nothing to compare against. The .so exists in scripts/_codemod_fast/ but is untracked.

9.3 Gate status — measured, with env -u PYTHONPATH

The structural gates are already green. Only formatting plus two genuine violations stand between the branch and gate-clean.

9.4 The two real violations — and a third the gates miss

1. T2 — FakeRepo at test_observability.py:555. A recording double for the repository Protocol, inside test_flush_with_repository. Straight RUL-002 violation. Fix, already in the design: plans/codemod-engine-design.md §2.1 specifies an InMemoryCodemodRepository in the persistence layer, and it was never built. Build it as a real adapter in src/ and use it. That is a genuine implementation, not a double, and it closes the gap the spec already anticipated.

2. T7 — pytest.mark.skip at test_workspace_load.py:220. RUL-002 violation.

3. Not caught by any gate, and the worst of the three — hardcoded worktree paths:

   test_workspace_load.py:17   FEATURE_PATH = "/home/mohamed/workspace/wt-QWEN/specs/.../workspace_load.feature"
   test_workspace_load.py:218  real_root    = Path("/home/mohamed/workspace/wt-QWEN")
   test_workspace_index.py:14  FEATURE_PATH = "/home/mohamed/workspace/wt-QWEN/specs/.../workspace_index.feature"
   

   Two of these are pytest-bdd scenarios() bindings. RUL-021 deletes wt-QWEN at merge. The moment it goes, those feature files are unreachable and the real_root fixture takes its skip branch. TST-110-001 — round-trip fidelity, the flagship "prove no silent corruption" test — stops running, silently, after the merge is declared done. This is precisely the surprise the investigation was meant to find. Resolve paths relative to the test file; add a bdd_features_base_dir; delete the skip so an unreachable tree is a failure, not a pass.

9.5 tree-sitter removal — smaller blast radius than feared

tree-sitter is a main dependency on master (not optional-extras) and is used by scripts/code-review.py::_check_imports_with_tree_sitter. Two corrections to my earlier concern:

• src/.../core/threading/test_executor.py matches tree_sitter only in a comment (the GIL_UNSAFE_MODULES quarantine note). The batched runner does not import it. No breakage.
• code-review.py guards with importlib.util.find_spec, so removal does not crash it — the deep-AST import check silently stops running. That file already carries a comment about this exact class of bug ("the tree-sitter path silently never ran even when installed").

Still restore the five deps and five wheel pins: a silently-degraded gate is the failure mode this project keeps paying for.

9.6 Suite integration — cheap

• Codemod tests are hermetic: tmp_path, real files, real sockets, real inotify. No Redis, no graph, no gateway. The only Neo4j/Redis strings in test_cli.py are literal fixture content for a replace pass.
• No pytest markers used at all, so master's requires_neo4j→requires_graph rename does not touch them.
• 16 new test modules join the 1460-module batched run. tests/refactor/codemod/__init__.py exists, so discovery works.
• .codemod_daemon/ is not in .gitignore. It holds the socket, pid and lock. Combined with the checkpoint hook (§6) — which commits whatever is in the tree — a running daemon can get its socket and pid file committed. Add the ignore entry before the daemon is ever started.

9.7 What I deliberately did not do

Run the codemod test suite. Opencode is actively editing those files; a run against a tree changing underneath produces a result that means nothing, and it would collide with its own runs. This is the single largest remaining unknown — no one has seen these 183 scenarios pass. It is the first action after opencode reports done, not something to infer now.

9.8 Ordered runbook

Each step is gated by the previous one.

Steps 1–7 are all runnable now, in parallel, inside wt-QWEN — none needs the merge. Nothing here has to wait for the last minute except 8–14.

10. Definition of done for the merge

master carries the SPEC-110 package, its tests, its spec/design/briefs and the new deps; all six disabled wheel pins are restored and the GIL is still off; every @traces(reqs=[...]) resolves against spec.md under one numbering scheme; the full suite is green from a non-master checkout; git rev-parse master and .git/config are unchanged by that run; nothing from another worktree rode along; and every commit is authored by Mohamed with no AI attribution.

Merge-complete is not integration-complete. Integration is done when the engine is wired (D6), each retirement in §7 has a parity case behind it, and the tools it replaces are gone — tracked separately, not gating the merge.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks