orchestrata_ultimate.refactor.ctok

Reference › orchestrata_ultimate › refactor › ctok › init

The facade of ctok-parser, the C tokenizer and lossless token layer of the SPEC-110 codemod engine. It re-exports the public names of tokenizer, token_stream, node_index, edit_buffer and lossless_module. The codemod workspace, server, passes, review modules and the scripts.codemod script import from here.

At a glance


Source	`src/orchestrata_ultimate/refactor/ctok/__init__.py`
Lines of code	37
Classes	0
Functions	0
Imports (internal)	`orchestrata_ultimate.refactor.ctok.edit_buffer`, `orchestrata_ultimate.refactor.ctok.lossless_module`, `orchestrata_ultimate.refactor.ctok.node_index`, `orchestrata_ultimate.refactor.ctok.token_stream`, `orchestrata_ultimate.refactor.ctok.tokenizer`
Imported by	`orchestrata_ultimate.refactor.codemod._parse_cache`, `orchestrata_ultimate.refactor.codemod.analysis.complexity`, `orchestrata_ultimate.refactor.codemod.passes`, `orchestrata_ultimate.refactor.codemod.report`, `orchestrata_ultimate.refactor.codemod.review_catalogue`, `orchestrata_ultimate.refactor.codemod.review_checks`, `orchestrata_ultimate.refactor.codemod.review_context`, `orchestrata_ultimate.refactor.codemod.review_engine`, `orchestrata_ultimate.refactor.codemod.server`, `orchestrata_ultimate.refactor.codemod.workspace`, `scripts.codemod`
Tests that import it	`tests/refactor/codemod/analysis/test_breaking_change.py`, `tests/refactor/codemod/analysis/test_read_only.py`, `tests/refactor/codemod/test_docstring_sweep_scope.py`, `tests/refactor/codemod/test_observability_wiring.py`, `tests/refactor/codemod/test_parse_cache.py`, `tests/refactor/codemod/test_review_checks.py`, `tests/refactor/codemod/test_review_coverage.py`, `tests/refactor/codemod/test_review_engine.py`, `tests/refactor/codemod/test_review_engine_commit_wiring.py`, `tests/refactor/codemod/test_review_passes.py`, `tests/refactor/codemod/test_transactional_writeback.py`, `tests/refactor/codemod/test_transform_passes.py`, `tests/refactor/codemod/test_workspace_load.py`

Class diagram

No classes; see Functions.

Dependency diagram

API

No public API: the module defines no classes, functions or constants.

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/refactor/ctok/__init__.py

"""ctok-parser: FT-safe C tokenizer + token-array lossless CST layer (SPEC-110).

Replaces LibCST's Rust parser for the SPEC-110 codemod engine
(``plans/contracts/ctok-parser.contract.json``, REQ-110-120 through
REQ-110-127). ``CTokenizer.tokenize`` drives a C extension (``scripts/_ctok/``)
that itself drives CPython's own battle-tested lexer, returning a
``TokenStream`` whose ``reconstruct()`` is byte-identical to the input source
by construction. ``LosslessModule`` marries that token array to an ``ast``
overlay via ``NodeIndex`` (P2): byte-identical ``.code`` (bytes), rename
targets locatable without libcst, and ``EditBuffer`` span-splice edits guarded
by the Black-derived ``verify_edit`` AST-equivalence check.
"""

from orchestrata_ultimate.refactor.ctok.edit_buffer import (
    EditBuffer,
    EditSafetyError,
    stringify_ast,
    verify_edit,
)
from orchestrata_ultimate.refactor.ctok.lossless_module import LosslessModule
from orchestrata_ultimate.refactor.ctok.node_index import NodeIndex, NodeSpanError
from orchestrata_ultimate.refactor.ctok.token_stream import TokenStream
from orchestrata_ultimate.refactor.ctok.tokenizer import CTokenizer, Token, TokenKind

__all__ = [
    "CTokenizer",
    "EditBuffer",
    "EditSafetyError",
    "LosslessModule",
    "NodeIndex",
    "NodeSpanError",
    "Token",
    "TokenKind",
    "TokenStream",
    "stringify_ast",
    "verify_edit",
]