orchestrata_ultimate.verification._non_python_scan

Reference › orchestrata_ultimate › verification › _non_python_scan

Collects non-Python implementation evidence for the implementation_scanner, its only importer. collect_non_python_source walks a project for JS/TS files (.js, .mjs, .jsx, .ts, .tsx) and HTML/CSS files (matched by basename only), skipping directories such as node_modules, .git, dist and build. For JS/TS it calls _run_js_ts_serializer, which launches scripts/serialize_js_ts_ast.py as a subprocess (resolved from the project root, not the working directory) with a timeout of 120 seconds by default, and _symbols_from_ast_json turns its JSON into NonPythonSymbol records. The result is a NonPythonSource.

Failures never raise. Every failure the call can observe (missing script, missing Node, non-zero exit, timeout, unreadable or invalid output) becomes a short diagnostic string, so the scanner can report a degraded scan instead of mistaking an unreadable source for nothing to find. A malformed JS/TS file is not detected as a diagnostic: it shows as a non-zero js_ts_file_count with no symbols.

At a glance

Class diagram

Dependency diagram

API

Constants

Classes

NonPythonSymbol (dataclass)

Decorators: dataclass(frozen=True) · Line 91

One JS/TS class or function, as extracted by the Node serializer.

NonPythonSource (dataclass)

Decorators: dataclass(frozen=True) · Line 103

Non-Python evidence ingredients for :class:ProjectSource.

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/verification/_non_python_scan.py

"""Non-Python evidence collection for :mod:`implementation_scanner`.

Restores the non-Python half of implementation evidence lost when the
scanner was rebuilt AST-precise around
:class:`orchestrata_ast.analyzer.CodebaseAnalyzer` (2026-08-12,
``plans/contracts/implementation-scanner-ast.contract.json``) --
``CodebaseAnalyzer.analyze()`` finds Python files ONLY
(``orchestrata_ast/analyzer.py:_find_python_files``), so every requirement
implemented in JS/TS/HTML/CSS silently read as a gap.

This module is the ONE seam that walks a filesystem for non-Python files --
mirrors ``implementation_scanner.build_project_source``'s own seam for the
Python half (``TestNoFilesystemWalk`` in the scanner's test suite proves
*that* module contains no ``os.walk``/``rglob``/``glob``/``open()`` of its
own; this module is where that responsibility for non-Python evidence
lives instead, kept out of the scanner's own source on purpose).

Coverage by extension (the 7 non-Python extensions master's regex scanner
collected, restored here AST-precisely where an AST layer exists):

- ``.js`` ``.mjs`` ``.jsx`` ``.ts`` ``.tsx`` -- AST-precise class/function
  evidence via ``scripts/serialize_js_ts_ast.py`` (a Node.js/TypeScript-
  compiler-API parse, shelled out to as a SUBPROCESS -- never imported as
  ``scripts.*``, which the contract above forbids from ``src/``).
- ``.html`` ``.css`` -- basename evidence ONLY. There is no AST layer for
  either in this codebase; pretending otherwise would silently under- or
  over-claim evidence, so this is a documented limitation, not an
  oversight. A future HTML/CSS AST source (if one is ever added) plugs in
  here without changing :mod:`implementation_scanner`'s detector shape.
- Every collected file (regardless of extension) also contributes its
  extension-less basename as evidence -- master's universal-basename rule,
  restored for parity with the Python side's own ``module-presence``
  detector (:func:`implementation_scanner._modules_of`), which already
  does this for ``.py`` files.

Node-toolchain diagnostics: whether the JS/TS serializer runs at all is
observable, not implicit. :func:`collect_non_python_source` invokes it
ONLY when at least one JS/TS-extension file was found by this module's own
walk; :data:`NonPythonSource.js_ts_diagnostic` is ``None`` in that "nothing
to parse" case. When JS/TS files ARE present but the subprocess fails --
missing ``node`` executable, a parse error, a timeout, or unreadable
output -- ``js_ts_diagnostic`` carries a short human-readable reason and
:data:`NonPythonSource.js_ts_symbols` is empty. Both cases return an empty
symbol tuple; only the diagnostic field tells them apart, which is what
lets a caller distinguish "no JS/TS files" from "could not parse JS/TS"
instead of reading both as an identical, silent zero.
"""

from __future__ import annotations

import json
import os
import subprocess
import sys
import tempfile
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path

from orchestrata_ultimate.core.paths import get_project_root

#: Extensions the Node/TypeScript-compiler-API serializer understands.
JS_TS_EXTENSIONS: tuple[str, ...] = (".js", ".mjs", ".jsx", ".ts", ".tsx")

#: Extensions with basename-only evidence -- no AST layer exists for them.
BASENAME_ONLY_EXTENSIONS: tuple[str, ...] = (".html", ".css")

#: The full non-Python evidence-relevant extension set this module collects.
NON_PYTHON_EXTENSIONS: tuple[str, ...] = JS_TS_EXTENSIONS + BASENAME_ONLY_EXTENSIONS

#: Directories never walked into. Mirrors ``serialize_js_ts_ast.js``'s own
#: ``SKIP_DIRS`` exactly (installed dependencies, VCS metadata, and build
#: output are never evidence, and walking ``node_modules`` on a real
#: generated project would otherwise turn every third-party package file
#: into a spurious basename-evidence hit).
_SKIP_DIRS: frozenset[str] = frozenset(
    {
        "node_modules",
        ".git",
        "__pycache__",
        "dist",
        "build",
        ".next",
        "coverage",
        ".ts-tmp",
    }
)


@dataclass(frozen=True)
class NonPythonSymbol:
    """One JS/TS class or function, as extracted by the Node serializer."""

    kind: str  # "class" | "function"
    name: str
    qualified_name: str
    file_path: str
    line: int
    is_async: bool = False


@dataclass(frozen=True)
class NonPythonSource:
    """Non-Python evidence ingredients for :class:`ProjectSource`.

    ``basenames`` is ``(stem, file_path)`` for every collected non-Python
    file. ``js_ts_symbols`` is empty in THREE distinct situations, and the
    caller must be able to tell them apart -- conflating them is the exact
    defect this module exists to close:

    1. No JS/TS files were found, so the serializer was never invoked.
       ``js_ts_file_count == 0`` and ``js_ts_diagnostic is None``.
    2. The toolchain could not run (missing script, missing/broken
       ``node``, non-zero exit, timeout, unreadable or malformed output).
       ``js_ts_diagnostic`` carries the reason.
    3. The serializer ran and reported success but extracted nothing from
       ``js_ts_file_count > 0`` files. ``js_ts_diagnostic is None``,
       because nothing FAILED -- this is what a file with no top-level
       symbols looks like, and also what a malformed file looks like
       after the TypeScript compiler error-recovers from it (measured:
       ``export class {{{`` exits 0 reporting ``Classes: 0, Functions:
       0``). The counts, not the diagnostic, are what expose it.

    ``js_ts_diagnostic`` therefore means "could not look", never "looked
    and found nothing" -- and it is only honest about case 3 in company
    with ``js_ts_file_count``.
    """

    basenames: tuple[tuple[str, str], ...] = ()
    js_ts_symbols: tuple[NonPythonSymbol, ...] = ()
    js_ts_diagnostic: str | None = None
    js_ts_file_count: int = 0


def _collect_non_python_files(root: Path) -> list[Path]:
    """Every JS/TS/HTML/CSS file under *root*, skipping ``_SKIP_DIRS``.

    The one directory walk this module exists to own (see the module
    docstring) -- kept out of :mod:`implementation_scanner` itself.
    """

    found: list[Path] = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [name for name in dirnames if name not in _SKIP_DIRS]
        here = Path(dirpath)
        found.extend(
            here / fname
            for fname in filenames
            if Path(fname).suffix in NON_PYTHON_EXTENSIONS
        )
    return sorted(found)


#: Names the TypeScript compiler emits for symbols that HAVE no name. An
#: anonymous default export (``export default function () {}``) serializes
#: as ``<anonymous>``; a malformed file the compiler error-recovers from
#: serializes as ``""``. Neither is an identifier a requirement can cite,
#: and emitting them puts ``implemented('SPEC', '')`` -- an empty Prolog
#: atom -- into the fact stream. Measured on real serializer output, not
#: assumed: see tests/verification/test_implementation_scanner.py.
_UNNAMED_SYMBOL_NAMES = frozenset({"", "<anonymous>"})

#: Cap on a single diagnostic line. A diagnostic exists to be READ on a
#: terminal; an unbounded one scrolls the summary it accompanies off the
#: screen, which defeats the point of surfacing it.
_DIAGNOSTIC_MAX_CHARS = 300


def _named(entry: Mapping[str, object]) -> bool:
    """Whether *entry* carries a usable identifier name.

    Guards the three symbol loops below: absent key, non-string value, and
    the compiler's placeholder names for genuinely unnamed symbols all
    resolve to ``False`` so they never reach ``EvidenceRecord.identifier``.
    """
    name = entry.get("name")
    if not isinstance(name, str):
        return False
    return name.strip() not in _UNNAMED_SYMBOL_NAMES


def _symbols_from_ast_json(data: Mapping[str, object]) -> list[NonPythonSymbol]:
    """Turn ``serialize_js_ts_ast.py``'s output JSON into flat symbols.

    Defensive on shape (``.get`` throughout): a malformed or
    schema-drifted document degrades to fewer symbols found, never a
    ``KeyError`` crash of the whole scan.
    """
    symbols: list[NonPythonSymbol] = []
    modules = data.get("modules", [])
    if not isinstance(modules, list):
        return symbols
    for module in modules:
        if not isinstance(module, dict):
            continue
        module_file_path = str(module.get("file_path", ""))
        for func in module.get("functions", []) or []:
            if not isinstance(func, dict) or not _named(func):
                continue
            symbols.append(
                NonPythonSymbol(
                    kind="function",
                    name=str(func["name"]),
                    qualified_name=str(func.get("qualified_name", func["name"])),
                    file_path=str(func.get("file_path", module_file_path)),
                    line=int(func.get("line", 0) or 0),
                    is_async=bool(func.get("is_async", False)),
                )
            )
        for cls in module.get("classes", []) or []:
            if not isinstance(cls, dict) or not _named(cls):
                continue
            symbols.append(
                NonPythonSymbol(
                    kind="class",
                    name=str(cls["name"]),
                    qualified_name=str(cls.get("qualified_name", cls["name"])),
                    file_path=str(cls.get("file_path", module_file_path)),
                    line=int(cls.get("line", 0) or 0),
                )
            )
            for method in cls.get("methods", []) or []:
                if not isinstance(method, dict) or not _named(method):
                    continue
                symbols.append(
                    NonPythonSymbol(
                        kind="function",
                        name=str(method["name"]),
                        qualified_name=str(
                            method.get("qualified_name", method["name"])
                        ),
                        file_path=str(method.get("file_path", module_file_path)),
                        line=int(method.get("line", 0) or 0),
                        is_async=bool(method.get("is_async", False)),
                    )
                )
    return symbols


def _run_js_ts_serializer(
    project_dir: Path,
    *,
    name: str,
    env: Mapping[str, str] | None,
    timeout: float,
    script_path: Path | None = None,
) -> tuple[list[NonPythonSymbol], str | None]:
    """Shell out to ``scripts/serialize_js_ts_ast.py`` (which shells out to
    Node itself) and turn its JSON output into symbols.

    *script_path* defaults to resolving from the repo root
    (:func:`get_project_root`), never relative to ``cwd`` -- this module
    may run from any working directory. The explicit-override seam
    (mirrors ``serialize_js_ts_ast.py``'s own ``_serialize(...,
    script_path=...)``) exists so the missing-script diagnostic branch is
    reachable in a test without relocating the real file on disk. Returns
    ``(symbols, diagnostic)``: *diagnostic* is ``None`` when the
    subprocess ran and its output was read, otherwise a short reason
    covering every failure THIS CALL can observe -- missing script,
    missing/broken ``node``, non-zero exit, timeout, unreadable output,
    invalid JSON.

    It does NOT cover a malformed source file: the TypeScript compiler
    error-recovers, exits 0, and reports ``Classes: 0, Functions: 0``, so
    this call cannot tell that from a file with no top-level symbols
    (measured -- ``export class {{{`` behaves exactly so). That case is
    exposed by ``NonPythonSource.js_ts_file_count`` being non-zero while
    ``js_ts_symbols`` is empty, not by a diagnostic. Promising otherwise
    here would be the write-only-comfort version of the very defect this
    module exists to close.
    """
    if script_path is None:
        script_path = get_project_root() / "scripts" / "serialize_js_ts_ast.py"
    if not script_path.is_file():
        return [], f"JS/TS serializer script not found: {script_path}"

    with tempfile.TemporaryDirectory(prefix="impl-scan-jsts-") as tmp:
        output_path = Path(tmp) / "ast.json"
        cmd = [
            sys.executable,
            str(script_path),
            "serialize",
            "--source",
            str(project_dir),
            "--name",
            name,
            "--output",
            str(output_path),
        ]
        try:
            result = subprocess.run(
                cmd,
                capture_output=True,
                text=True,
                env=dict(env) if env is not None else None,
                timeout=timeout,
            )
        except subprocess.TimeoutExpired:
            return [], f"JS/TS serializer timed out after {timeout}s"
        except OSError as exc:
            return [], f"JS/TS serializer could not be launched: {exc}"

        if result.returncode != 0:
            # The LAST non-empty line, not a tail of five: when the child
            # dies on a Python traceback the final line is the exception
            # ("FileNotFoundError: ... 'node'") and the four above it are
            # frame noise that buries it in an operator's stderr.
            diagnostic_lines = [
                line.strip()
                for line in (result.stderr or result.stdout or "").splitlines()
                if line.strip()
            ]
            tail = (
                diagnostic_lines[-1][:_DIAGNOSTIC_MAX_CHARS]
                if diagnostic_lines
                else f"exit code {result.returncode}"
            )
            return [], f"JS/TS serializer failed: {tail}"

        try:
            raw_text = output_path.read_text()
        except OSError as exc:
            return [], f"JS/TS serializer output unreadable: {exc}"

    try:
        data = json.loads(raw_text)
    except json.JSONDecodeError as exc:
        return [], f"JS/TS serializer produced invalid JSON: {exc}"

    if not isinstance(data, dict):
        return [], "JS/TS serializer produced a non-object JSON document"

    return _symbols_from_ast_json(data), None


def collect_non_python_source(
    project_dir: Path,
    *,
    name: str,
    env: Mapping[str, str] | None = None,
    timeout: float = 120.0,
) -> NonPythonSource:
    """Compose a :class:`NonPythonSource` for *project_dir*.

    The one entry point :func:`implementation_scanner.build_project_source`
    composes for the non-Python half of evidence -- it walks *project_dir*
    once (:func:`_collect_non_python_files`), then invokes the JS/TS
    serializer ONLY when the walk found at least one JS/TS-extension file
    (never unconditionally: an all-HTML/CSS or all-Python project must
    never pay for or diagnose a Node invocation it has no use for).

    Args:
        project_dir: Root directory to scan (already validated by the
            caller -- this function does not check it exists).
        name: Project name, forwarded to the JS/TS serializer for its own
            diagnostics.
        env: Optional explicit environment for the JS/TS serializer
            subprocess (the injection seam tests use to exercise a broken
            Node toolchain without monkeypatching); ``None`` inherits the
            current process environment, the production default.
        timeout: Seconds to wait for the JS/TS serializer subprocess.
    """
    files = _collect_non_python_files(project_dir)
    if not files:
        return NonPythonSource()

    basenames = tuple((path.stem, str(path)) for path in files)
    js_ts_count = sum(1 for path in files if path.suffix in JS_TS_EXTENSIONS)
    if not js_ts_count:
        return NonPythonSource(basenames=basenames)

    symbols, diagnostic = _run_js_ts_serializer(
        project_dir, name=name, env=env, timeout=timeout
    )
    return NonPythonSource(
        basenames=basenames,
        js_ts_symbols=tuple(symbols),
        js_ts_diagnostic=diagnostic,
        js_ts_file_count=js_ts_count,
    )