Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › codemod-engine-implementation-summary

SPEC-110: In-Memory Code Intelligence Platform — Implementation Summary

Source: plans/codemod-engine-implementation-summary.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-27 · 215 lines.

Prose pending.

Plan text

SPEC-110: In-Memory Code Intelligence Platform — Implementation Summary

Version: 2.0.0
Date: 2026-07-27
Status: Complete (post-adversarial review, ready for implementation)

Executive Summary

SPEC-110 defines a complete in-memory code intelligence platform with daemon architecture that loads an entire Python codebase as a lossless Concrete Syntax Tree (CST) forest, builds a multihop-queryable Code Intelligence Graph (CIG), executes composable transform and review passes entirely in RAM, and commits changes atomically with full rollback capability. The platform operates as a long-running daemon process (eliminating cold-start cost), communicates via Unix socket IPC using JSON-RPC 2.0, watches for file changes via inotify, and persists provenance to a graph database. This is not merely a codemod engine — it is a code intelligence substrate exploiting Python 3.14.6 free-threaded (GIL disabled, 12 cores) to achieve 3x parallel parse speedup, O(1) graph queries via CSR adjacency arrays, and 40-check in-memory review without subprocess overhead.

Architecture Overview

The platform is organized as a 10-layer system operating in three phases:

Phase 1: Load & Index (RAM-only)

  1. CST Forest — LibCST 1.8.6 eager parse of all .py files into lossless Module trees. Parallel parse via ThreadPoolExecutor (12 cores). Byte-perfect round-trip: module.code.encode('utf-8') == on_disk_bytes. ~3 GiB RSS for 2,500 files (780K LOC, 30 MB on disk).

  2. Code Intelligence Graph (CIG) — In-memory property graph with nodes (Module, Class, Function, Method, Parameter, Test, Import) and edges (IMPORTS, CALLS, INHERITS, DEFINES, TESTED_BY, OVERRIDES). Supports multihop queries: blast radius, breaking changes, cycles (Tarjan SCC), topological order (Kahn), centrality, coupling, impact score.

  3. CSR Index — Compressed Sparse Row adjacency arrays (array.array('i')) for O(1) dependency/dependent lookups. Forward and transpose matrices for bidirectional queries. BFS transitive closure for blast radius computation.

Phase 2: Transform & Review (RAM-only)

  1. Transform Passes — 5 built-in passes: ReplaceStringLiteralPass, AddImportPass, RemoveUnusedImportPass, DocstringSweepPass, RenameSymbolPass (SPEC-109 interop). Per-file parallel dispatch via ThreadPoolExecutor for embarrassingly parallel workloads. Passes compose: pass N sees pass N-1's mutations.

  2. Review Passes — 40 checks in 7 categories (complexity, style, quality, structure, safety, correctness, test isolation). 4 built-in review passes: ParseIntegrity, BannedConstructs, DiffBudget, PrologSemanticReview. All operate on mutated CST trees — verdicts before bytes.

Phase 3: Commit (disk I/O)

  1. Transactional Write-Back — Atomic commits via tempfile + os.rename. Batch fsync (5 directory fds instead of 100 file fds, 20x I/O reduction). Post-write re-parse verification. Full rollback on any failure. Concurrent-edit guard: mtime + xxhash3_64 content hash + O_NOFOLLOW + fcntl.flock.

Cross-Cutting Layers

  1. Observability — 5 observers on SyncObservable[CodemodEvent]: TracingObserver (structured JSON spans), GraphWriterObserver (async batched writes to graph DB), ErrorDetector (pattern alerts via EventBus), MetricsObserver (Prometheus counters/histograms), DebugPanelObserver (NDJSON trace files). Log-and-continue policy for observer errors.

  2. Graph Persistence — Repository pattern (CodemodGraphRepository Protocol) with BoltSyncCodemodRepository implementation. Cypher MERGE patterns for CodemodRun, CodemodPass, CodemodFinding, CodeChange nodes. New ontology labels and relationships extending canonical_schema.py.

  3. Prolog Semantic Review — codemod_review.pl predicates: pass_safe/3, blast_radius_ok/3, import_cycle_free/2, rename_safe/3, string_replace_safe/3. Asserts mutations as Prolog facts, queries safety predicates, uses explain:explain_whynot/2 for proof trees. Via janus-swi vendored free-threaded fork.

  4. Daemon Architecture — Long-running service keeps workspace in memory (3 GiB RSS) between invocations. Unix socket at .codemod_daemon/socket for IPC. JSON-RPC 2.0 protocol. inotify file watching for incremental reload (<500ms). Lifecycle: codemod daemon start|stop|status|restart. Single daemon per repo (lock file). Multiple concurrent clients (parallel reads, serialized writes).

Wave Execution Plan

Wave Name Agent Model Duration Status Description
W-1 Contract Authoring Claude Opus 4.6 ~30 min DONE Mermaid classDiagram, contract JSON, Prolog predicates
W0 Scaffold SCAFFOLD Haiku ~15 min Pending Skeleton generation, types, conftest, libcst+xxhash deps
W1 Forest + Sentinel FOREST + SENTINEL Qwen Max + Sonnet ~20 min Pending Workspace, CIG, CSR, txn (parallel agents)
W2 Architect + Reviewer ARCHITECT + REVIEWER Qwen Max + Sonnet ~20 min Pending Transform passes, analysis, review, observability (parallel)
W3 Integrator INTEGRATOR Qwen Max ~15 min Pending CLI, SPEC-109 interop, benchmark tests
W4 Persistence PERSISTENCE Qwen Max ~15 min Pending Repository, graph writer, ontology extensions
W5 C Extension C-EXT Qwen Max ~20 min Pending _codemod_fast sub-package (CSR, xxhash, mmap in C)
W6 Quality QUALITY Claude ~15 min Pending Gates, coverage, skeleton-diff, final report

Total estimated time: ~2.5 hours (with parallel waves)

Agent roster: 6 agents, 3 models (Haiku for scaffold, Qwen Max for implementation, Sonnet for parallel tracks, Claude for quality)

Key Design Decisions

ID Decision Rationale
D1 LibCST over stdlib ast Byte-perfect round-trip; preserves comments, whitespace, formatting
D2 Whole-module serialization CST writes modules, not text spans; simplifies rollback
D3 Reviews on mutated forest Verdicts before bytes; catches errors in RAM before disk damage
D4 mtime+hash concurrent-edit guard TOCTOU prevention; xxhash3_64 at 5 GB/s; O_NOFOLLOW + flock
D5 No I/O in passes Write-back exclusively via commit(); enforced by construction
D6 No registry/volk wiring Deferred (same policy as SPEC-109); keeps engine self-contained
D7 CSR over dict-of-sets O(1) queries, contiguous memory, 10x faster for BFS transitive closure
D8 Daemon architecture Long-running service eliminates cold-start cost; enables interactive use
D9 Batch fsync 5 directory fsyncs instead of 100 file fsyncs; 20x I/O reduction
D10 Prolog semantic review Leverages existing codebase_db.pl + explain.pl; declarative safety checks
D11 Gherkin BDD for test specification Living documentation; behavioral specification; tag-based traceability
D12 C extension via janus-swi pattern Py_MOD_GIL_NOT_USED declaration; Python fallback ensures portability

Performance Targets

# Metric Target Measurement
1 Daemon startup <125s 120s parse + 5s socket setup
2 IPC round-trip (query) <100ms Unix socket + JSON-RPC
3 IPC round-trip (transform) <10s Unix socket + pass execution
4 File change detection <500ms inotify event + incremental re-parse
5 Concurrent clients 10+ Parallel read operations
6 Full-repo load (serial) <360s 2,500 files, 780K LOC, single-threaded
7 Full-repo load (parallel) <120s 12 cores, ThreadPoolExecutor
8 Per-file pass (parallel) <10s 2,500 files, 12 cores
9 Write-back (100 files) <50ms Batch fsync, atomic rename
10 Dependency query <2ms CSR index, O(1) lookup
11 Blast radius (3 hops) <10ms BFS over CSR
12 Prolog semantic review <500ms Assert facts, query predicates
13 RSS delta <3 GiB Full codebase (30 MB disk → ~500-600 MB CST → ~3 GiB RSS)
14 In-memory review (40 checks) <1s LibCST trees, zero subprocess calls

File Manifest

Source Files

Category Count Key Files
Core modules 12 workspace.py, report.py, passes.py, review.py, txn.py, _cig.py, _cig_builder.py, _csr_index.py, __init__.py
Analysis modules 7 blast_radius.py, breaking_changes.py, complexity.py, coverage.py, abstraction.py, coupling.py, __init__.py
Observability modules 6 events.py, tracing.py, graph_writer.py, error_detector.py, metrics.py, debug_panel.py (+ __init__.py)
Persistence modules 3 repository.py, bolt_repository.py, __init__.py
Daemon modules 4 daemon.py, server.py, client.py, watcher.py
Prolog 1 codemod_review.pl
CLI script 1 scripts/codemod.py
C extension sub-package 1 scripts/_codemod_fast/ (6 files: csr_index.c, content_hash.c, mmap_loader.c, module.c, setup.py, pyproject.toml)

Total: ~5,500 LOC Python + ~700 LOC C

Test Files

Category Count Details
Gherkin feature files 16 126 scenarios across workspace_load, workspace_index, transform_passes, review_before_write, transactional_writeback, cli, spec109_interop, code_intelligence_graph, observability, graph_persistence, analysis_passes, in_memory_review, daemon_lifecycle, ipc_protocol, file_watching, concurrency
Test files 15 141 total tests (BDD scenario bindings + plain pytest for benchmarks, CSR unit tests, C extension parity)

Contracts and Briefs

File Purpose
plans/contracts/spec-110-codemod-engine.mmd Mermaid classDiagram contract
plans/contracts/spec-110-codemod-engine.contract.json Machine-readable metadata (classes, reuse, wiring, test plan)
plans/codemod-engine-briefs/BRIEF-W0.md through BRIEF-W5.md Agent task briefs for each wave

Adversarial Review Findings

GPT-4.1 Review

GPT-4.1 identified 20 findings (4 CRITICAL, 5 HIGH, 4 MEDIUM, 7 LOW). Key outcomes:

Severity Finding Verdict Action Taken
CRITICAL LibCST parallelism bottleneck Partially valid Added benchmark gate to Wave 0
CRITICAL C extension build fragility Already mitigated Python fallback pattern; no change needed
CRITICAL Prolog process-global bottleneck Wrong Reviewer lacked project context (vendored FT fork)
CRITICAL TOCTOU race conditions Valid Added O_NOFOLLOW + fcntl.flock to txn.py
HIGH Graph DB write bottleneck Valid Added batching + buffer-overflow-drop to GraphWriterObserver
HIGH No support for large repos Out of scope Added explicit non-goal (repos >10K files)

Fable Review

Fable review validated the overall architecture and identified additional edge cases in the observer error handling and CSR incremental update paths, which were absorbed into the design.

Plan Amendments (Post-Review)

  1. Wave 0 benchmark gate for LibCST parse speed validation
  2. O_NOFOLLOW + fcntl.flock added to concurrent-edit guard
  3. GraphWriterObserver batching + buffer-overflow-drop semantics
  4. Explicit non-goal: repos >10K files
  5. Threat model section added (path traversal, symlink attacks, TOCTOU, Cypher injection)
  6. Hypothesis property-based tests deferred to Phase 2

Success Criteria

The implementation is complete when all of the following are met:

  1. All requirements (REQ-110-001 through REQ-110-140) are implemented and traced with @traces
  2. All 14 acceptance tests (TST-110-001 through TST-110-014) pass
  3. All 126 BDD scenarios across 16 feature files pass
  4. All 141 total tests pass (BDD + plain pytest)
  5. Coverage >=90% per file
  6. All gates pass: ruff, black, standards, check-deps, test-review, pyright_ratchet
  7. Performance targets met (parse time, query latency, RSS delta, IPC round-trip)
  8. Graph ontology additions committed to canonical_schema.py
  9. C extension builds and passes parity tests (C/Python identical results)
  10. Prolog semantic review integrates with codemod_review.pl
  11. Observer architecture emits all 14 event types
  12. CLI accepts all subcommands (load-bench, run, report, analyze, daemon start/stop/status/restart)
  13. SPEC-109 interop verified (RenameSymbolPass consumes WorkspaceEditPlan)
  14. Transactional write-back passes all rollback and concurrent-edit guard tests
  15. 40-check in-memory review executes without subprocess calls
  16. Code Intelligence Graph supports all multihop queries in <2ms
  17. Daemon lifecycle: start, parse workspace, create socket, serve requests, graceful shutdown

Next Steps

With the specification, design document, contracts, feature files, and agent briefs all complete, the next step is to begin Wave 0 (Scaffold) implementation:

  1. Run uv add libcst xxhash to add required dependencies
  2. Run gen-skeleton on the Mermaid classDiagram contract to generate typed skeletons
  3. Split monolithic output into per-module files (report.py, workspace.py, etc.)
  4. Write conftest.py with shared BDD fixtures and step definitions
  5. Gate 0: Verify every skeleton class imports and constructs
  6. Commit: SPEC-110: scaffold — skeletons, types, conftest, libcst+xxhash deps

Then proceed through Waves 1-6 per the execution plan, with parallel agent dispatch where indicated.

This document summarizes the complete SPEC-110 implementation plan. For full details, see: