scripts.build_ft_tree_sitter

Reference › scripts › build_ft_tree_sitter

Builds free-threaded wheels of the tree-sitter stack (SPEC-023 GIL-off doctrine), so the AST tooling can run on the free-threaded interpreter that build_ft_cpython produces. A human runs it; tests/scripts/test_build_ft_tree_sitter.py imports it.

Invocation: main with no options builds every package in PACKAGES in a temporary directory and lists the wheels. --install then force-reinstalls them with uv pip install and verifies the stack. --verify only checks the tree-sitter packages currently installed (verify). Without --install it prints how to install the wheels through the project's normal dependency sync.

Each Package describes one sdist, with sdist_name and wheel_path. build_package downloads the source distribution from PyPI (_pypi_sdist_url, _download), checking its sha256 (_sha256) against a pinned value, fetches the grammar headers where needed (_grammar_headers), and builds the wheel with a pinned system toolchain (_pin_system_toolchain, _build_env).

At a glance

Class diagram

Dependency diagram

No internal imports and no importers.

API

Classes

Package (dataclass)

Decorators: dataclass · Line 105

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/scripts/build_ft_tree_sitter.py

#!/usr/bin/env python3
"""Build free-threaded tree-sitter wheels (SPEC-023 GIL-off doctrine).

Upstream tree-sitter packages cannot be installed on Python 3.14t as-is:

* ``tree-sitter`` (core) 0.26.0 **explicitly declares**
  ``PyUnstable_Module_SetGIL(module, Py_MOD_GIL_USED)`` at the end of
  ``PyInit__binding`` — importing it re-enables the GIL process-wide,
  silently undoing SPEC-023. The patch flips it to ``Py_MOD_GIL_NOT_USED``
  (the vendored-janus judgment: our usage is single-threaded parsing).
* Grammar sdists carry a ``bdist_wheel.get_tag`` override that retags every
  wheel ``cp310-abi3`` even when the limited API was disabled for the
  free-threaded build — the built wheel is then rejected as incompatible.
* Older grammar templates (cpp, typescript) hard-code ``py_limited_api=True``
  + a ``Py_LIMITED_API`` macro (the limited API does not exist on 3.14t) and
  ship a single-phase ``PyInit__binding`` with no GIL declaration.
* Several sdists are INCOMPLETE — ``tree-sitter-python`` omits
  ``src/scanner.c`` (undefined ``..._external_scanner_create`` at import),
  cpp/typescript omit ``src/tree_sitter/*.h`` and ``common/scanner.h``.
  Upstream never notices because their PyPI wheels are CI-built from git.
  Missing files are fetched from the immutable git TAG and sha256-verified.

This script makes the rebuild deterministic on a fresh clone:

1. download each PINNED sdist from PyPI and verify its sha256;
2. fetch sha256-PINNED missing files from the matching upstream git tag;
3. apply the source patches described above;
4. build each wheel with the project venv's python (``uv build --wheel
   --no-build-isolation``, so the extension sees ``Py_GIL_DISABLED``) into
   ``vendor/wheels/``;
5. ``--verify`` imports every language, parses a snippet, and asserts the
   GIL is still disabled.

``pyproject.toml`` points ``[tool.uv.sources]`` at the built wheels; the
wheels themselves are untracked (janus doctrine: binaries rebuild, never
commit). If ``uv sync`` fails with a missing-wheel path error, run:

    python scripts/build_ft_tree_sitter.py && uv sync --dev

Bump the version/sha256 pins together when upgrading; drop a package's
patches once upstream ships free-threaded (cp314t) wheels for it.
"""

from __future__ import annotations

import argparse
import hashlib
import os
import subprocess
import sys
import tarfile
import tempfile
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parent.parent
WHEEL_DIR = REPO_ROOT / "vendor" / "wheels"
VENV_PYTHON = REPO_ROOT / ".venv" / "bin" / "python3"

# Host-shell variables that leak a foreign toolchain's headers/libraries into
# the C compile. The Guix profile on the dev fleet exports these; with
# C_INCLUDE_PATH set, the compiler picks the Guix glibc headers and dies on
# the GCC-only _Float64 typedefs in bits/floatn.h (measured: Clang 19 vs
# /home/*/.guix-profile/include/math.h), and LIBRARY_PATH misdirects the
# linker the same way. Same hazard and remedy as scripts/build_janus_ft.py.
_BUILD_ENV_STRIP = ("C_INCLUDE_PATH", "CPLUS_INCLUDE_PATH", "CPATH", "LIBRARY_PATH")


def _pin_system_toolchain(env: dict[str, str]) -> dict[str, str]:
    """``worktree_toolchain.pin_system_toolchain``, loaded by file path so it
    works however this script was imported (the same seam worktrees.py uses):
    CC/CXX come from the ``env/toolchain.env`` declaration, unconditionally —
    never from PATH — and a missing, bare-named or non-executable declaration
    raises ``ToolchainDeclarationError``, so the build is refused."""
    import importlib.util

    spec = importlib.util.spec_from_file_location(
        "build_ft_tree_sitter_toolchain",
        Path(__file__).resolve().parent / "worktree_toolchain.py",
    )
    assert spec and spec.loader
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.pin_system_toolchain(env)


def _build_env() -> dict[str, str]:
    """Child environment for the C compile: the host shell minus the foreign
    toolchain contamination above, plus the DECLARED compiler: CC/CXX come
    from the ``env/toolchain.env`` declaration via
    ``worktree_toolchain.pin_system_toolchain``, unconditionally, never from
    PATH (a Guix gcc on PATH carries glibc headers with no Linux kernel
    headers; measured 2026-10-05: `linux/limits.h: No such file or
    directory`). A missing, bare-named or non-executable declaration raises
    ``ToolchainDeclarationError`` and the build is refused."""
    env = {k: v for k, v in os.environ.items() if k not in _BUILD_ENV_STRIP}
    return _pin_system_toolchain(env)


RAW = "https://raw.githubusercontent.com/tree-sitter"


@dataclass
class Package:
    name: str
    version: str
    sdist_sha256: str
    # repo-relative extra files missing from the sdist: {dest: (url, sha256)}
    fetch: dict[str, tuple[str, str]] = field(default_factory=dict)
    # (path, old, new) exact-string source patches; every one must apply
    patches: list[tuple[str, str, str]] = field(default_factory=list)

    @property
    def sdist_name(self) -> str:
        return f"{self.name.replace('-', '_')}-{self.version}"

    @property
    def wheel_path(self) -> Path:
        return WHEEL_DIR / (f"{self.sdist_name}-cp314-cp314t-linux_x86_64.whl")


# The abi3 retag override shared by every grammar setup.py template.
_RETAG_OLD = '        if python.startswith("cp"):'
_RETAG_NEW = (
    '        if python.startswith("cp")' ' and not get_config_var("Py_GIL_DISABLED"):'
)

# Single-phase PyInit in the older grammar template (cpp / typescript).
_INIT_OLD = (
    "PyMODINIT_FUNC PyInit__binding(void) {\n"
    "    return PyModule_Create(&module);\n"
    "}"
)
_INIT_NEW = (
    "PyMODINIT_FUNC PyInit__binding(void) {\n"
    "    PyObject *m = PyModule_Create(&module);\n"
    "#ifdef Py_GIL_DISABLED\n"
    "    if (m != NULL) {\n"
    "        PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);\n"
    "    }\n"
    "#endif\n"
    "    return m;\n"
    "}"
)

_HDR = {  # shared src/tree_sitter headers (identical across both repos/tags)
    "parser.h": "a1f6ef161fbaf48a0e10fca90ef5290a062462b307b3898aa562993853b9f80a",
    "alloc.h": "b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea",
    "array.h": "4ff743903dc46f5db6aa54f31c6b4d160a8a9779e5b2ab1ee59ae7ebcd850ea1",
}


def _grammar_headers(
    repo: str, tag: str, src_dirs: list[str]
) -> dict[str, tuple[str, str]]:
    out: dict[str, tuple[str, str]] = {}
    for d in src_dirs:
        for h, digest in _HDR.items():
            out[f"{d}/tree_sitter/{h}"] = (
                f"{RAW}/{repo}/{tag}/{d}/tree_sitter/{h}",
                digest,
            )
    return out


PACKAGES = [
    Package(
        name="tree-sitter",
        version="0.26.0",
        sdist_sha256="b40c219edccc4564530c96f8f1556f6202b37cda964d1cbd7bd2b7e68b40a245",
        patches=[
            (
                "tree_sitter/binding/module.c",
                "PyUnstable_Module_SetGIL(module, Py_MOD_GIL_USED);",
                "PyUnstable_Module_SetGIL(module, Py_MOD_GIL_NOT_USED);",
            ),
        ],
    ),
    Package(
        name="tree-sitter-python",
        version="0.25.0",
        sdist_sha256="b13e090f725f5b9c86aa455a268553c65cadf325471ad5b65cd29cac8a1a68ac",
        fetch={
            "src/scanner.c": (
                f"{RAW}/tree-sitter-python/v0.25.0/src/scanner.c",
                "6db82134ac2d4c90a1a1475487a625cface02662ebda9b7478cad9c7147e9afe",
            ),
        },
        patches=[("setup.py", _RETAG_OLD, _RETAG_NEW)],
    ),
    Package(
        name="tree-sitter-c",
        version="0.24.2",
        sdist_sha256="1628584df0299b5a340aa63f8e67b6c97c91517f52fa7e7a4c557e40adb330a9",
        # 0.24.2 already has the FT-aware limited-api logic, the fixed retag
        # guard, the Py_mod_gil slot, and no external scanner — build as-is.
    ),
    Package(
        name="tree-sitter-cpp",
        version="0.23.4",
        sdist_sha256="6a59c4cebb1ad1dc2e8d586cf8a72b39d21b8108b7b139d089719e81a339e41d",
        fetch=_grammar_headers("tree-sitter-cpp", "v0.23.4", ["src"]),
        patches=[
            (
                "setup.py",
                "from setuptools import",
                "from sysconfig import get_config_var\nfrom setuptools import",
            ),
            ("setup.py", _RETAG_OLD, _RETAG_NEW),
            (
                "setup.py",
                "py_limited_api=True,",
                'py_limited_api=not get_config_var("Py_GIL_DISABLED"),',
            ),
            ("setup.py", '                ("Py_LIMITED_API", "0x03090000"),\n', ""),
            ("bindings/python/tree_sitter_cpp/binding.c", _INIT_OLD, _INIT_NEW),
        ],
    ),
    Package(
        name="tree-sitter-typescript",
        version="0.23.2",
        sdist_sha256="7b167b5827c882261cb7a50dfa0fb567975f9b315e87ed87ad0a0a3aedb3834d",
        fetch={
            **_grammar_headers(
                "tree-sitter-typescript",
                "v0.23.2",
                ["typescript/src", "tsx/src"],
            ),
            "common/scanner.h": (
                f"{RAW}/tree-sitter-typescript/v0.23.2/common/scanner.h",
                "da66ef2bd14a3f7ea743e25ba068c6c9aae2c3509db200ff80c4a0e6116e564c",
            ),
        },
        patches=[
            (
                "setup.py",
                "from setuptools import",
                "from sysconfig import get_config_var\nfrom setuptools import",
            ),
            ("setup.py", _RETAG_OLD, _RETAG_NEW),
            (
                "setup.py",
                "py_limited_api=True,",
                'py_limited_api=not get_config_var("Py_GIL_DISABLED"),',
            ),
            ("setup.py", '                ("Py_LIMITED_API", "0x03090000"),\n', ""),
            (
                "bindings/python/tree_sitter_typescript/binding.c",
                _INIT_OLD,
                _INIT_NEW,
            ),
        ],
    ),
]

VERIFY_SNIPPET = """
import sys
import tree_sitter_c, tree_sitter_cpp, tree_sitter_python, tree_sitter_typescript
from tree_sitter import Language, Parser
cases = [
    ("python", tree_sitter_python.language(), b"import os"),
    ("c", tree_sitter_c.language(), b"int main(void) { return 0; }"),
    ("cpp", tree_sitter_cpp.language(), b"class A { int x; };"),
    ("typescript", tree_sitter_typescript.language_typescript(), b"const x: number = 1;"),
    ("tsx", tree_sitter_typescript.language_tsx(), b"const el = <div>hi</div>;"),
]
for name, lang, src in cases:
    tree = Parser(Language(lang)).parse(src)
    assert not tree.root_node.has_error, f"{name}: parse FAILED"
    print(f"{name}: parse ok ({tree.root_node.type})")
if sys._is_gil_enabled():
    raise SystemExit("FAIL: the GIL is enabled — a wheel lost its patch")
print("GIL: off — all five languages free-threaded clean")
"""


def _sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def _download(url: str, dest: Path, sha256: str) -> None:
    with urllib.request.urlopen(url, timeout=120) as resp:
        dest.write_bytes(resp.read())
    got = _sha256(dest)
    if got != sha256:
        raise SystemExit(f"sha256 mismatch for {url}: got {got}, want {sha256}")


def _pypi_sdist_url(pkg: Package) -> str:
    import json

    url = f"https://pypi.org/pypi/{pkg.name}/{pkg.version}/json"
    with urllib.request.urlopen(url, timeout=60) as resp:
        data = json.load(resp)
    for u in data["urls"]:
        if u["packagetype"] == "sdist":
            return u["url"]
    raise SystemExit(f"no sdist listed on PyPI for {pkg.name}=={pkg.version}")


def build_package(pkg: Package, work: Path) -> Path:
    tarball = work / f"{pkg.sdist_name}.tar.gz"
    print(f"[{pkg.name}] downloading sdist {pkg.version}")
    _download(_pypi_sdist_url(pkg), tarball, pkg.sdist_sha256)
    with tarfile.open(tarball) as tf:
        tf.extractall(work, filter="data")
    src = work / pkg.sdist_name

    for rel, (url, digest) in pkg.fetch.items():
        dest = src / rel
        dest.parent.mkdir(parents=True, exist_ok=True)
        print(f"[{pkg.name}] fetching missing {rel} from upstream tag")
        _download(url, dest, digest)

    for rel, old, new in pkg.patches:
        target = src / rel
        text = target.read_text()
        if old not in text:
            raise SystemExit(f"[{pkg.name}] patch pattern not found in {rel}")
        target.write_text(text.replace(old, new, 1))
        print(f"[{pkg.name}] patched {rel}")

    WHEEL_DIR.mkdir(parents=True, exist_ok=True)
    print(f"[{pkg.name}] building wheel (no build isolation, 3.14t)")
    subprocess.run(
        [
            "uv",
            "build",
            "--wheel",
            "--no-build-isolation",
            "-o",
            str(WHEEL_DIR),
            str(src),
        ],
        check=True,
        env=_build_env(),
    )
    if not pkg.wheel_path.exists():
        raise SystemExit(f"[{pkg.name}] expected wheel missing: {pkg.wheel_path.name}")
    return pkg.wheel_path


def verify() -> int:
    proc = subprocess.run([str(VENV_PYTHON), "-c", VERIFY_SNIPPET])
    return proc.returncode


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Build free-threaded tree-sitter wheels (SPEC-023)."
    )
    parser.add_argument(
        "--verify",
        action="store_true",
        help="only verify the currently installed tree-sitter stack",
    )
    parser.add_argument(
        "--install",
        action="store_true",
        help="uv pip install the built wheels after building",
    )
    args = parser.parse_args()

    if args.verify:
        return verify()

    with tempfile.TemporaryDirectory(prefix="ft-tree-sitter-") as tmp:
        wheels = [build_package(pkg, Path(tmp)) for pkg in PACKAGES]

    print("\nBuilt wheels:")
    for w in wheels:
        print(f"  {w.relative_to(REPO_ROOT)}")

    if args.install:
        subprocess.run(
            ["uv", "pip", "install", "--force-reinstall"] + [str(w) for w in wheels],
            check=True,
        )
        return verify()
    print(
        "\nInstall + verify: uv sync --dev  (wheels are wired in pyproject)"
        "\nor: python scripts/build_ft_tree_sitter.py --install"
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())