orchestrata_ultimate.refactor.ctok

Reference › orchestrata_ultimate › refactor › ctok

ctok-parser: FT-safe C tokenizer + token-array lossless CST layer (SPEC-110).

ctok-parser is the lossless source layer of the SPEC-110 codemod engine: it lets the engine rewrite Python files at the byte level without disturbing anything it did not mean to change. The codemod workspace, server, passes and review modules in codemod import it through the facade, and the codemod and ctok_roundtrip_gate scripts use it directly.

The parts are layered. tokenizer wraps the C tokenizer and yields tokens; token_stream holds those tokens with the original bytes and can reconstruct them exactly; node_index maps ast nodes to token spans; edit_buffer accumulates span edits and checks them against an AST-equivalence gate; and lossless_module combines the stream, the tree and the index into one immutable object whose edits return a new object.

Data flows from bytes to tokens to a parsed module with an index; an edit names a token span, is spliced into the bytes and the result is reparsed. Start reading at lossless_module, then edit_buffer.

6 modules, 829 lines of code.

Modules

Dependency diagram