scripts.ctok_roundtrip_gate

Reference › scripts › ctok_roundtrip_gate

The ctok round-trip gate, the binary acceptance test for the ctok-parser contract (P1). It checks that tokenizing every Python file in the repository with the CTokenizer from tokenizer and then reconstructing the file gives back the same bytes. tests/scripts/test_ctok_roundtrip_gate.py imports it.

Invocation: main takes an optional root (default the repository root) and has no command-line options. _iter_py_files lists the Python files. For each file, main reads the bytes, tokenizes and reconstructs them, and records a mismatch if the file is unreadable, if tokenizing or reconstructing raises, or if the bytes differ, with the on-disk and reconstructed sizes. It prints how many files were byte-identical and lists the mismatched files on standard error. The result is 0 only if every file round-trips.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

API

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/scripts/ctok_roundtrip_gate.py

#!/usr/bin/env python3
"""ctok round-trip gate — P1's binary acceptance for the ctok-parser contract.

Parses every ``.py`` file under ``src/``, ``scripts/``, ``tests/`` through
:class:`~orchestrata_ultimate.refactor.ctok.tokenizer.CTokenizer` and asserts
``TokenStream.reconstruct() == <on-disk bytes>`` (REQ-110-120,
``plans/contracts/ctok-parser.contract.json``). Prints
``N/N byte-identical, M mismatches`` and exits non-zero on any mismatch —
this is P1's binary gate: nothing downstream may trust the token layer
until this is green tree-wide.

Usage:
    uv run python scripts/ctok_roundtrip_gate.py
"""

from __future__ import annotations

import sys
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parent.parent
INCLUDE_DIRS = ("src", "scripts", "tests")


def _iter_py_files(root: Path) -> list[Path]:
    """Every ``.py`` file under the include roots, sorted for stable output.

    Args:
        root: Repository root directory.

    Returns:
        Sorted list of absolute paths to ``.py`` files.
    """
    files: list[Path] = []
    for subdir in INCLUDE_DIRS:
        subdir_path = root / subdir
        if subdir_path.exists():
            files.extend(subdir_path.rglob("*.py"))
    return sorted(files)


def main(root: Path | None = None) -> int:
    """Run the round-trip gate over the whole tree.

    Args:
        root: Repository root to scan; defaults to :data:`REPO_ROOT`.

    Returns:
        0 if every file round-trips byte-identically, 1 otherwise.
    """
    from orchestrata_ultimate.refactor.ctok.tokenizer import CTokenizer

    root = REPO_ROOT if root is None else root
    files = _iter_py_files(root)
    total = len(files)
    mismatches: list[tuple[Path, str]] = []

    for path in files:
        try:
            on_disk = path.read_bytes()
        except OSError as exc:
            mismatches.append((path, f"unreadable: {exc}"))
            continue
        try:
            stream = CTokenizer.tokenize(on_disk)
            reconstructed = stream.reconstruct()
        except (
            Exception
        ) as exc:  # noqa: BLE001 - any tokenizer failure is a gate finding
            mismatches.append((path, f"tokenize/reconstruct raised: {exc!r}"))
            continue
        if reconstructed != on_disk:
            mismatches.append(
                (
                    path,
                    f"byte mismatch: {len(on_disk)} on-disk bytes vs "
                    f"{len(reconstructed)} reconstructed bytes",
                )
            )

    ok_count = total - len(mismatches)
    print(f"{ok_count}/{total} byte-identical, {len(mismatches)} mismatches")

    if mismatches:
        print("\nMismatched files:", file=sys.stderr)
        for path, reason in mismatches:
            rel = path.relative_to(root)
            print(f"  {rel}: {reason}", file=sys.stderr)
        return 1

    return 0


if __name__ == "__main__":
    raise SystemExit(main())