Knowledge Wiki Orchestrata Ultimate

Free-Threaded Prolog — janus-swi under Python 3.14t

STATUS: PRODUCTION (graduated 2026-07-18, session 242). This is not a proof-of-concept. The patched fork is the installed, default, fail-fast Prolog query transport of orchestrata-ultimate: uv sync builds and installs it from scripts/_janus_patched ([tool.uv.sources]), CodebaseTools uses it as THE transport and raises if janus is missing or the GIL is on (no fallback, user ruling), the release wheel is hosted on GitLab (janus-swi-ft 1.5.3-orchestrata.1, installable anywhere via install_janus_ft.sh --from-gitlab), and the full 26,859-test authoritative suite runs green through it. Reverting to stock janus is FORBIDDEN.

Python: 3.14.6 free-threading build (python3.14t, Py_GIL_DISABLED, GIL genuinely off — no PYTHON_GIL=0) SWI-Prolog: 10.x (libswipl.so.10) janus-swi: 1.5.3, vendored + patched at scripts/_janus_patched/

Goal

Run the governance / tabling Prolog paths under Python 3.14t with the GIL genuinely off (not forced via PYTHON_GIL=0) and zero crashes, so the project keeps its free-threaded parallelism instead of falling back to a process-wide GIL every time janus_swi is imported.

An undeclared C extension re-enables the process-wide GIL the moment it is imported under a free-threaded interpreter. Stock janus-swi 1.5.3 does exactly that, and — once the GIL is off — additionally crashes on the tabling restraint path. Both had to be fixed at the C level in the extension itself.

The crash, visually

Before the individual bugs, the shape of the worst one (bug 2). Under free-threading there is no GIL to implicitly attach a thread state, so when Prolog calls back into Python on a thread that released its state, the Python allocator has no per-thread heap to work with:

Diagram projects-orchestrata-ultimate-free-threaded-prolog-1

The other two bugs are the quieter siblings of the same theme — state the GIL used to serialize implicitly (the module hashtable, the deferred-decref queue) suddenly shared without a lock. The sections below give each one its root cause, fix, and reproducer.

Three distinct bugs, all in the free-threaded (Py_GIL_DISABLED) paths

1. GIL re-enabled on import (module never declared FT-safe)

Single-phase PyModule_Create + PyUnstable_Module_SetGIL did not suppress CPython's process-wide GIL re-enable. The fix is multi-phase init in mod_swipl.c: PyInit__swipl returns PyModuleDef_Init(&swipl_module) with a Py_mod_exec slot and, critically:

static PyModuleDef_Slot swipl_module_slots[] = {
  {Py_mod_exec, swipl_module_exec},
  {Py_mod_multiple_interpreters, Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED},
  {Py_mod_gil, Py_MOD_GIL_NOT_USED},
  {0, NULL}
};

Now import janus_swi leaves the GIL off with no env var.

Build trap. uv/setuptools build isolation compiles a GIL-re-enabling .so — the isolated build backend picks a non-free-threaded interpreter, so Py_GIL_DISABLED is undefined and the multi-phase declaration compiles out. The package MUST be built --no-build-isolation against this project's interpreter. This is wired in pyproject.toml ([tool.uv] no-build-isolation-package = ["janus-swi"]).

2. Missing PyThreadState on SWI→Python callbacks (deterministic SEGV)

The prior patch replaced PyGILState_Ensure() with a bare mutex lock in the free-threaded py_gil_ensure. But PyGILState_Ensure() is not a no-op under free-threading — it attaches a PyThreadState (and thus this thread's mimalloc heap). swipl_next_solution runs PL_next_solution inside Py_BEGIN_ALLOW_THREADS, which detaches the tstate; when SWI then calls back into Python (py_call → py_eval → PyObject_GetAttrString → PyUnicode_FromString → _PyObject_MiMalloc) the thread has no tstate → NULL per-thread heap → deterministic crash on the undefined-answer / restraint tabling path (ownership_chain/3, max_answers(20), cyclic graph). AddressSanitizer pinpointed _PyObject_MiMalloc dereferencing a NULL heap.

Fix (janus.c): the free-threaded py_gil_ensure/py_gil_release attach and detach a real tstate exactly as the GIL build does for foreign-called threads:

static int py_gil_ensure(py_gil_state *state)
{ if ( !py_init() ) return FALSE;
  state->gil = PyGILState_Ensure();       /* attaches tstate + mimalloc heap */
  if ( delayed ) delayed_decref(NULL);
  return TRUE; }

static void py_gil_release(py_gil_state state)
{ if ( delayed ) delayed_decref(NULL);
  PyGILState_Release(state.gil); }

MyPy_DECREF gates on the real PyGILState_Check(). The prior global janus_mutex was removed here (non-recursive → would deadlock the re-entrant Prolog↔Python paths). Deterministic reproducer: 8/8 crash → 0/N.

3. Unprotected shared C state once the GIL was gone (rare race)

Removing the GIL exposed process-global C state the GIL used to serialize. An audit found two structural offenders:

Lazy-initialized singletons (enum_constructor, Fraction/Term ctors, the janus module ref, pred/user, the hex borrowed ref) are hardened under the same lock (SGLOCK/SGUNLOCK), plus an explicit Py_XINCREF(hex).

The -O0-masked-bug lesson

Bug #2 is timing-fragile: an unoptimised (-O0 / ASAN) build happens to leave a usable tstate attached and does not crash, so an -O0 "validation" is a false green. The bug only reliably reproduces under the interpreter's real -O3 flags — which is exactly what a normal uv sync build uses. Always validate the release (-O3) binary, i.e. the one uv sync actually installs, never a hand-built debug .so.

Reproducible build

The fork is vendored (source only — scripts/_janus_patched/.gitignore excludes build/, *.so, *.egg-info) and wired as a path source:

# pyproject.toml
[tool.uv]
no-build-isolation-package = ["janus-swi"]

[tool.uv.sources]
janus-swi = { path = "scripts/_janus_patched" }

setuptools/wheel are in the dev dependency group so the no-isolation build has its backend. uv sync --dev builds the extension against this project's free-threaded interpreter and installs it.

Local rebuild gotcha. setuptools does incremental builds: a stale scripts/_janus_patched/build/ from an earlier compile can relink a pre-patch object file and silently produce a crashing .so. A fresh clone is clean (the dir is git-ignored); when rebuilding in place after editing the C, rm -rf scripts/_janus_patched/build first, or uv sync --reinstall-package janus-swi.

Verify

# GIL genuinely off, no env var:
env -u PYTHON_GIL .venv/bin/python -c \
  "import sys, janus_swi; print('GIL enabled:', sys._is_gil_enabled())"   # -> False

Scope: the whole hot path is now GIL-off (msgpack + grpcio RESOLVED)

This work makes janus GIL-safe: importing janus_swi (and the prolog_adapter / governance Prolog path) leaves the GIL off, and the tabling crash is gone. The two other hot-path C extensions that used to re-enable the process-wide GIL have both since been fixed (this section supersedes the earlier "msgpack is the next domino" note — see docs/free-threaded-prolog.md in-repo for the authoritative version):

Regression guard: tests/conftest.py imports both grpc and msgpack and fails the suite if sys._is_gil_enabled() goes True — so a dependency regression cannot silently re-enable the GIL.

GIL-flip census & the full extension fleet (session 251, 2026-07-21)

A full census (import every C-extension package in the venv in a fresh python3.14t subprocess, check sys._is_gil_enabled()) found exactly three remaining GIL-flippers: tree_sitter, tokenizers, httptools. They were the root cause of order-dependent batch test failures: a co-batched test importing one of them permanently enabled the GIL for that pytest subprocess, and every janus-transport test dispatched after it failed its free-threading assertion — passing solo, failing in batches.

Resolution, per offender:

Verify the stack any time: python scripts/build_ft_tree_sitter.py --verify (parses all five languages and asserts the GIL is still off).

Approaches explored and abandoned

Two throwaway exploration scripts lived under scripts/ during this work. Both are recorded here and then deleted — their scratch code (path hacks, sub-standard imports, no tests) does not belong in the tree, but the findings do.

Verification performed

Release packaging & universal installer (session 242, 2026-07-18)

The patched fork is now a distributable release, not just an in-repo build:

CodebaseTools janus transport wiring (session 242)

The patched janus is now the default query transport for the Prolog codebase-intelligence layer (src/orchestrata_ultimate/tools/codebase_tools.py over codebase_db.pl):

Still open: registering CodebaseTools in the volk ToolRegistry (which volks, and the production reindex() trigger) — tracked in the orchestrata session backlog.