orchestrata_ultimate.core.janus_bootstrap

Reference › orchestrata_ultimate › core › janus_bootstrap

A preload that makes SWI-Prolog's library(json) work under janus-swi. The cause, documented in the module and in the free-threaded Prolog page, is symbol visibility: Python loads C extensions with local visibility, so when SWI later loads its JSON foreign extension, that extension cannot see PL_release_stream in janus's private copy of libswipl. The fix is to load libswipl with RTLD_GLOBAL through ctypes before janus_swi is imported. ensure_swipl_symbols_global does this and returns the path it loaded; the module's janus attribute is the import point that code uses in place of importing janus_swi directly. prolog_adapter is the main consumer.

_resolve_libswipl_path picks the library in a fixed order (ADR-0091): the ORCHESTRATA_LIBSWIPL override; the janus extension's own ldd linkage (_janus_extension_path locates the extension, _path_from_janus_ldd reads its link); then the runtime variables of the SWI-Prolog declared in env/toolchain.env (_declared_swipl, _path_from_dump_runtime_variables); otherwise RuntimeError. There is no hardcoded path and no bare swipl from PATH; the module records that a hardcoded library path once stopped the production kernel on a host where that file did not exist.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

API

Constants

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/core/janus_bootstrap.py

# mypy: ignore-errors
"""janus_bootstrap.py — RTLD_GLOBAL preload that fixes ``library(json)`` under janus.

Root cause (proven, not speculative — see ``docs/free-threaded-prolog.md``
§"The PL_release_stream/library(http/json) ABI caveat"): the failure is
**not** a missing symbol. ``json.so`` (SWI's ``library(json)`` foreign
extension) needs ``PL_release_stream``, and the declared ``libswipl.so``
genuinely exports it — ``janus_swi``'s compiled ``_swipl`` extension links
exactly that library (confirmed via ``ldd``; on this host the Guix 9.3.8
``libswipl.so.9`` that ``env/toolchain.env`` declares). The failure is
**RTLD_LOCAL symbol visibility**: Python loads C extensions (including
``janus_swi._swipl``) with ``RTLD_LOCAL``, so when janus pulls in
``libswipl`` its symbols never land in the process's *global* dynamic
scope. When SWI later ``dlopen``s ``json.so`` to satisfy a lazy
``use_module(library(json))``, that extension's own (also-``RTLD_LOCAL``)
symbol lookup for ``PL_release_stream`` can't see back into janus's
private copy of libswipl and fails.

The fix: explicitly preload ``libswipl`` with
``ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL)`` **before** ``janus_swi`` is
imported. That puts libswipl's symbols in the global scope, so every
SWI foreign extension loaded afterwards — including ``json.so`` — resolves
against it. Verified empirically: ``atom_json_dict/3`` and
``json_read_dict/3`` now run to completion in-process under janus with
this preload in place; without it, they raise the ``PL_release_stream``
error every time.

Which libswipl that is, is resolved in a fixed order (see
``_resolve_libswipl_path``; ADR-0091): the ``ORCHESTRATA_LIBSWIPL``
override, then the janus extension's OWN ``ldd`` linkage — authoritative,
because its RUNPATH is the library directory of the SWI-Prolog declared in
``env/toolchain.env`` that the fork baked at build time — then
``--dump-runtime-variables`` of that DECLARED swipl, and finally a
``RuntimeError``. There is no hardcoded path and no bare ``swipl`` from
PATH: the prod kernel died on 2026-10-11 preloading a hardcoded
``/usr/lib/swipl/.../libswipl.so.10`` that does not exist on this host,
after the ``/gnu/store`` RUNPATH its janus was built with had been
collected by ``guix gc``; the test fleet's own ``/usr/lib/swipl`` 10.x is
not this extension's backend either.

Usage — anywhere ``janus_swi`` would otherwise be imported directly:

    from orchestrata_ultimate.core.janus_bootstrap import janus

This module resolves the libswipl path, performs the preload exactly once
per process (idempotent, thread-safe), and re-exports the real
``janus_swi`` module as ``janus`` — so callers get the identical API
surface, just with the ABI caveat closed.
"""

import ctypes
import importlib.util
import logging
import os
import re
import shutil
import subprocess
import threading
from collections.abc import Callable, Mapping
from pathlib import Path

logger = logging.getLogger(__name__)

_PLLIBSWIPL_RE = re.compile(r'^(\w+)="([^"]*)"')

_preload_lock = threading.Lock()
_preloaded_path: str | None = None


def _declared_swipl(env: Mapping[str, str]) -> str | None:
    """The SWI-Prolog ``env/toolchain.env`` declares, or ``None``.

    Only an absolute, executable path qualifies: a bare ``swipl`` is NEVER
    looked up on PATH (ADR-0091). Under the systemd PATH
    (``/usr/local/bin:/usr/bin``) there is no swipl at all, and in the test
    fleet's ``app`` container the one on PATH is a 10.x whose home is not the
    backend this extension was built against — preloading it alongside a
    ``libswipl.so.9`` extension SIGABRTs.
    """
    declared = env.get("SWIPL") or ""
    if not declared or not os.path.isabs(declared):
        return None
    if not os.access(declared, os.X_OK):
        logger.warning(
            "SWIPL=%s from the environment is not an executable; ignoring it "
            "(env/toolchain.env declares the SWI-Prolog, ADR-0091)",
            declared,
        )
        return None
    return declared


def _path_from_dump_runtime_variables(swipl_bin: str | None = None) -> str | None:
    """Resolve libswipl via ``<swipl_bin> --dump-runtime-variables``
    (PLLIBSWIPL, falling back to PLBASE/PLARCH/PLSOEXT if that key is absent).

    ``swipl_bin`` is the DECLARED swipl (``_declared_swipl``); ``None`` or an
    empty string resolves nothing, because there is no PATH lookup any more
    (ADR-0091 — a bare ``swipl`` is whichever build the environment happens to
    carry). The parameter stays injectable so tests can exercise every branch
    of this resolver deterministically (monkeypatch/PATH mutation is banned
    suite-wide; see AGENTS.md).
    """
    if not swipl_bin:
        return None
    swipl = swipl_bin
    try:
        result = subprocess.run(
            [swipl, "--dump-runtime-variables"],
            capture_output=True,
            text=True,
            timeout=10,
            check=True,
        )
    except OSError, subprocess.SubprocessError:
        return None

    variables: dict[str, str] = {}
    for line in result.stdout.splitlines():
        m = _PLLIBSWIPL_RE.match(line.strip())
        if m:
            variables[m.group(1)] = m.group(2)

    direct = variables.get("PLLIBSWIPL")
    if direct and Path(direct).exists():
        return direct

    plbase = variables.get("PLBASE")
    plarch = variables.get("PLARCH")
    plsoext = variables.get("PLSOEXT", "so")
    if plbase and plarch:
        libdir = Path(plbase) / "lib" / plarch
        # The ABI soname suffix is NOT part of --dump-runtime-variables and
        # differs across distributions (libswipl.so.10 on the Debian-ish
        # reference install, libswipl.so.9 on the Guix 9.3.8 the dev/test
        # fleet runs) — pick the newest versioned libswipl actually present.
        candidates = sorted(
            (p for p in libdir.glob(f"libswipl.{plsoext}.*") if p.suffix[1:].isdigit()),
            key=lambda p: int(p.suffix[1:]),
            reverse=True,
        )
        if candidates:
            return str(candidates[0])
    return None


def _janus_extension_path(
    package_name: str = "janus_swi", locations: list[str] | None = None
) -> str | None:
    """Locate janus-swi's compiled ``_swipl*.so`` extension WITHOUT importing
    ``janus_swi`` (the whole point is to preload before that import).

    ``package_name``/``locations`` override package/module discovery --
    exposed so tests can exercise the "package not found" and "no compiled
    extension in this location" branches deterministically without
    installing/uninstalling anything.
    """
    if locations is None:
        try:
            pkg_spec = importlib.util.find_spec(package_name)
        except ImportError, ValueError, ModuleNotFoundError:
            return None
        if pkg_spec is None or not pkg_spec.submodule_search_locations:
            return None
        locations = list(pkg_spec.submodule_search_locations)
    for location in locations:
        matches = sorted(Path(location).glob("_swipl*.so"))
        if matches:
            return str(matches[0])
    return None


def _path_from_janus_ldd(
    ext_path: str | None = None, ldd_bin: str | None = None
) -> str | None:
    """Resolve libswipl via the janus extension's own ``ldd`` linkage — the
    file it will actually dlopen, so this is authoritative when it works: the
    fork bakes the DECLARED install's library directory (``env/toolchain.env``'s
    ``SWIPL``, ADR-0091) as the extension's RUNPATH, so this resolves under any
    PATH — including a systemd unit's — and survives ``guix gc``.

    ``ext_path``/``ldd_bin`` override extension-path discovery and the
    ``shutil.which("ldd")`` lookup -- same testability rationale as
    ``_path_from_dump_runtime_variables``.
    """
    if ext_path is None:
        ext_path = _janus_extension_path()
    if ext_path is None:
        return None
    ldd = shutil.which("ldd") if ldd_bin is None else (ldd_bin or None)
    if ldd is None:
        return None
    try:
        result = subprocess.run(
            [ldd, ext_path],
            capture_output=True,
            text=True,
            timeout=10,
            check=True,
        )
    except OSError, subprocess.SubprocessError:
        return None
    for line in result.stdout.splitlines():
        if "libswipl" not in line or "=>" not in line:
            continue
        _, _, rhs = line.partition("=>")
        candidate = rhs.strip().split(" ", 1)[0].strip()
        if candidate and Path(candidate).exists():
            return candidate
    return None


def _resolve_libswipl_path(
    env: Mapping[str, str] | None = None,
    dump_runtime_resolver: Callable[[str | None], str | None] = (
        _path_from_dump_runtime_variables
    ),
    ldd_resolver: Callable[[], str | None] = _path_from_janus_ldd,
) -> str:
    """Resolve the libswipl shared-object path to preload, in priority order:

    1. ``ORCHESTRATA_LIBSWIPL`` env override (explicit escape hatch).
    2. the janus extension's own ``ldd`` linkage — AUTHORITATIVE, because it
       is the file the extension actually loads: the fork bakes the declared
       install's library directory as its RUNPATH (ADR-0091), so this resolves
       under any PATH and survives ``guix gc``.
    3. ``--dump-runtime-variables`` of the DECLARED swipl only
       (``env/toolchain.env``'s ``SWIPL`` — never a bare name from PATH).
    4. nothing resolved → :class:`RuntimeError` naming the declaration and the
       rebuild. There is no hardcoded fallback path: the one this function
       used to return (``/usr/lib/swipl/lib/x86_64-linux/libswipl.so.10``)
       does not exist on this host, and preloading it is what killed the prod
       kernel on 2026-10-11 (ADR-0091).

    ``env`` defaults to ``os.environ`` and is exposed so tests can exercise
    the override without mutating the process environment (matching
    ``core/infra_config.py``'s ``_resolve_bin`` convention).
    ``dump_runtime_resolver``/``ldd_resolver`` default to the real strategy
    functions above; overriding them lets tests force the LATER tiers (tier 3
    and tier 4's RuntimeError) without breaking the real swipl install on this
    machine -- tier 2 succeeds here for any extension built against the
    declaration, so those branches are otherwise unreachable in this
    environment. ``dump_runtime_resolver`` is handed the declared swipl (or
    ``None`` when the declaration is unusable), which is what keeps a bare
    ``SWIPL=swipl`` from ever being executed.
    """
    env = env if env is not None else os.environ
    override = env.get("ORCHESTRATA_LIBSWIPL")
    if override:
        return override

    path = ldd_resolver()
    if path:
        return path

    path = dump_runtime_resolver(_declared_swipl(env))
    if path:
        return path

    raise RuntimeError(
        "No libswipl resolved, so library(json)/library(http) cannot work "
        "under janus: the janus extension's own `ldd` linkage names no "
        "libswipl (its RUNPATH does not resolve — a stale build against a "
        "collected /gnu/store item?) and the environment declares no usable "
        "SWIPL. Declare SWIPL=/abs/path/to/bin/swipl in env/toolchain.env "
        "(ADR-0091) and rebuild the fork so its RUNPATH is that declared "
        "install's library directory: "
        "`uv sync --reinstall-package janus-swi`. ORCHESTRATA_LIBSWIPL "
        "overrides the resolution for a one-off start."
    )


def ensure_swipl_symbols_global(path_override: str | None = None) -> str:
    """Preload libswipl with ``RTLD_GLOBAL`` so its symbols (notably
    ``PL_release_stream``) are visible to foreign extensions SWI dlopens
    LATER, e.g. ``json.so`` for a lazy ``use_module(library(json))``.

    Idempotent and thread-safe (double-checked locking) — safe to call from
    any number of threads/import sites; only the first call actually does
    the ``ctypes.CDLL`` preload. Returns the resolved libswipl path.

    ``path_override`` bypasses ``_resolve_libswipl_path()`` entirely --
    production call sites (including the module-level bootstrap call below)
    never pass it; it exists so tests can exercise the ``ctypes.CDLL``
    failure branch with a deliberately-bad path without needing to break
    the real swipl install to do it.

    Raises:
        RuntimeError: no libswipl resolved, or the resolved path could not be
            dlopen'd. Per project doctrine (see ``core/prolog_adapter.py``,
            ``tools/codebase_tools.py``), this fails loud rather than silently
            leaving json/http broken.
    """
    global _preloaded_path
    if _preloaded_path is not None:
        return _preloaded_path
    with _preload_lock:
        if _preloaded_path is not None:
            return _preloaded_path
        path = path_override if path_override is not None else _resolve_libswipl_path()
        try:
            ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL)
        except OSError as exc:
            raise RuntimeError(
                f"Failed to RTLD_GLOBAL-preload libswipl at {path!r} -- "
                "this is required for library(json)/library(http) to work "
                "under janus (see docs/free-threaded-prolog.md's "
                "PL_release_stream/library(http/json) ABI caveat). Declare the "
                "SWI-Prolog in env/toolchain.env and rebuild the fork "
                "(ADR-0091: `uv sync --reinstall-package janus-swi`), or set "
                "ORCHESTRATA_LIBSWIPL to the right libswipl for a one-off "
                "start."
            ) from exc
        logger.info("Preloaded libswipl (RTLD_GLOBAL) from %s", path)
        _preloaded_path = path
        return path


# Preload BEFORE importing janus_swi -- the entire point of this module.
ensure_swipl_symbols_global()

import janus_swi as janus  # noqa: E402  -- must follow the RTLD_GLOBAL preload

__all__ = ["janus", "ensure_swipl_symbols_global"]