Orchestrata Ultimate › Research

Research — index

This is the index of the project's investigations. Each one asked a question, measured something, and came back with findings and a recommendation.

Every page says how it was measured and what is still open. A finding here is not a decision: decisions live in decisions, and the ones still waiting on a ruling are on open decisions.

Status words used below:

• Open: the question or its recommendation awaits a ruling or a measurement.
• Ruled: Mohamed has decided; the work may still be pending.
• Acted on: the recommendation is built or deployed.

New this session (2026-10-22/23)

Python deployment best practices. This page sets the PyPA, uv and PEP guidance against the repo. The guidance: one edited source per package, the wheel as the deployment artifact, editable installs for development only, one workspace with one lock, deptry in CI, and a strangler cut-over. The repo today has nine per-package locks, no workspace over packages/, an editable /opt/codetools dependency, and generated code that imports codetools.rtm. It presents both sides of the open dispute over whether packages/ is a deployment copy or the edited source.

codetools versus the components branch. codetools is the toolchain and the nine component packages are its output: they were generated by codetools realize and they test through codetools.rtm. Neither can replace the other. Merging components takes several things: the INT-01 to INT-25 increments, removing src/orchestrata_ultimate, one workspace and lock, deptry, porting 13 drifted files, and a codetools release pin.

Cut-over review and test re-point. The INT plan predates the wave tool that landed the other eight packages. 52% of its lines are test tooling. Two of its gates are already approved but still listed as open, and it has no increment for removing the old tree. Of 1,596 test modules that import orchestrata_ultimate, only 360 re-point by the same module path; the rest need a frozen-to-landed symbol map that does not yet exist.

Host inventories. This is a census of the host. It covers the deployments and their commits, the systemd units (all orchestrata units disabled since 2026-10-21), the Incus fleets, and the MCP servers (orchestrata's is broken). It counts 119 hook commands, of which 83 are live in a $HOME session, and the skills that load in each kind of session. It also lists the defects found, such as hooks that a config deploy deletes, hooks that never fire, and duplicated skills.

Hidden AST spawners and the editable-venv flip. Three background spawners run AST work, two of them from the wrong tree. A plain uv run in a hook re-syncs the venv as editable, and that made the item-3 push take four attempts. Separately, the Claude config deploy's rsync --delete wiped three hand-installed hooks. The fix is 8a2cb42ad (uv run --no-sync), which is not yet pushed.

Gateway token analysis. Every orchestrata MCP call fails with GATEWAY_TOKEN is not set. ADR-0099 expects a token from the deployed environment, but gateway tokens are one-hour JWTs. Three options are compared. B, where the transport mints its own JWT from the deployed signing secret, is recommended and awaits a ruling.

DeepChat history review. This page records the DeepChat sessions' technical work:

• the codetools Claude Code plugin;
• the work-machine install plan, which janus-swi blocked;
• the dependency-injection ruling, implemented as a5c419f5;
• the 2026-10-21 Memgraph benchmark, btrfs diagnosis, and shutdown of the fleets and 16 units;
• 48 untracked skills in wt-COMPONENTS.

Earlier research

Research digest 2026-07-21. This page holds the verdicts of six parallel investigations from sessions 242–244, plus two deploys. Four verdicts:

• DSL benefit: no. Every candidate surface fails the same way, with several hand-kept copies of one truth. Single-sourcing plus parity tests fixes that, and a grammar would only add another copy.
• Prolog next-impact: a ranked proposal.
• Module versioning: a workspace split is ruled out by the import graph.
• BDD runtime tracing for the RTM: partial yes.

It also records the dead-governance-rules bug fixed in 88e88797, the TRACES_TO pipeline deploy, and the RAG stack evaluation.

Graph engine analysis. A live-measured comparison of the two production graph engines of July 2026: Neo4j Community (no HA, GDS installed but unused) and the orchgraph Memgraph fork (Raft HA, in-memory). It covers capability, memory and migration cost. It records the MAGE build and deploy (from 16 to 140 procedures), and it surveys open-source alternatives. Neither engine won both roles.

Graph DB best practices. Ten findings from a 106-agent deep-research run, checked by adversarial refutation. Examples: property predicates need a label to use an index, PROFILE beats EXPLAIN for real costs, and size the page cache to the store. On Memgraph, the 1 s storage.access_timeout_sec default is too tight under concurrent access; it was raised to 60 s at all three layers. The page also records the deployment status of each finding.

RAG context-injection investigation. Written by DeepChat on 2026-10-21. Retrieval is split-brained: the full hybrid stack (vector, keyword, fusion, rerank) is built and tested but not wired, while production context injection runs a keyword-only CONTAINS loop. No retrieval path filters on project_id, so cross-project leakage is real but bounded to a path and one line of text. There are two fixes: route through the vector path, and scope retrieval by project.

Semantic graph QA benchmark. The 100-question RAG benchmark over the orchestrata codebase: questions, scoring keywords, coverage and the top-3 retrieved chunks. The snapshot on the page is the 2026-07-08 hybrid run, at 57% average keyword coverage with 32 low-quality answers. The page says the current state after the TESTS-edge enrichment is 82% with 0 low-quality answers.

Measurement retention and rule induction. The page sorts what is built, in progress and only proposed for the measurement instrument (SPEC-113). It shows why the prediction half of a "learning" Prolog layer was refuted, and it sets out the 2026-08-19 SPEC-114 proposal for quarantined rule induction with a "why-not" explainer. The proposal is mostly unbuilt.

Free-threaded Prolog. The production record of the patched janus-swi fork under Python 3.14t with the GIL genuinely off. It covers the three free-threading bugs that were fixed, the reproducible build, the GitLab-hosted wheel, the census of every extension module's GIL behaviour, and the wiring of the fork as the fail-fast Prolog transport. The full suite runs green through it.

Free-threaded Prolog POC (moved). A redirect page. The capability graduated from proof-of-concept to production, so its content lives on the Free-threaded Prolog page.

The Prolog layer does not learn. An audit of three easily confused claims. Facts rebuilt from snapshots: live. Facts accumulated from runtime observation: live and queryable. Rules learned from outcomes: absent, and rejected by decision with numbers attached. The recommendation was to build the measuring instrument, not a learning subsystem.

How the suite got 32% faster. The narrative of session 242. Profiling first showed that the minutes lived in boring phases, not in the glamorous ones. Five measured levers then took the authoritative 26,859-test suite from about 15.1 to about 10.2 minutes with zero skips. A wave-2 section shows that test teardowns, not connections, were the per-test tax.

Suite parallelism floor. Why the 30.6K-test gate floors at about 250 s on the 6-core/12-thread dev box with -j 14 --batch-multiplier 2. It gives the verdict on every lever tried. Rejected: -b 1, backfill on its own, 16 workers. Kept: mega-module splits and the LPT cache repair. Root-fixing every flake was the biggest saver, because each solo retry is serial time.

The components swarm and the deterministic pivot. On the night of 2026-10-14/15, a nine-hour frontier-model swarm produced about 32 classes, with overhead at 10–50 times the work. A 250-line script then reproduced the committed lift byte-for-byte. That proved the lift is a script, not an LLM task, and led to ADR-0103: an ephemeral byte-faithful oracle plus a separate clean target, gated by parity.

Related

• Open decisions
• Decisions
• Session 271
• Blockers
• Upcoming work
• Tests
• Project home