Orchestrata Ultimate › Plans › prolog-codeintel-verification-plan

Prolog codebase-intelligence verification plan (session 242)

Source: plans/prolog-codeintel-verification-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-18 · 103 lines.

Prose pending.

Plan text

Prolog codebase-intelligence verification plan (session 242)

Subject: commit 3b159679 "Add Prolog codebase intelligence layer" (author Hannah hannah@openclaw.ai — external, landed on master unreviewed, no tests shipped). Files: src/orchestrata_ultimate/tools/codebase_db.pl (521 LOC), src/orchestrata_ultimate/tools/codebase_tools.py (289 LOC), schema/ast-ultimate.schema.json, PROLOG_STATUS.md.

User directive: "Prolog code intelligence is better than Python for recursive work and supposedly faster — test and verify that." Deliverable = a numbers-backed PASS/FAIL verdict, plus the layer brought up to project standards (reviewed, fixed, tested).

Facts established (session 242 recon, this machine)

• swipl 10.0.2 on host (PROLOG_STATUS claims "Janus unavailable on SWI-Prolog 9.0.4" — wrong here: the project's patched free-threaded janus_swi imports fine in the 3.14t venv; it is the deployed governance Prolog path).
• plans/ast-ultimate.json present (12 MB, 2026-07-15): 695 modules, 7857 symbols, 3916 resolved call/inherit edges, 8 import cycles.
• Smoke test (explicit KB path): reindex 2.1 s, warnings []; stats/impact/ cycles correct-looking; ~1.2 s per query (each subprocess call re-consults the 10 MB facts file; the 0.3 s claim in PROLOG_STATUS is wrong on this data size).
• Qnames are package-root-relative (agents.volks.base.BaseAgent, no orchestrata_ultimate. prefix).

Defects already found (to be confirmed/extended by reviewer)

The benchmark (core user ask)

Three-way comparison on identical recursive workloads over the same graph:

• P-sub — Prolog via subprocess swipl (as shipped): cold, includes consult.
• P-janus — Prolog via in-process patched janus_swi: consult rules+facts once, then repeated queries; report first-call (table build) and memoized-call separately. If janus genuinely cannot consult/table this KB, report that as a finding — do not fake.
• Py-mem — pure-Python recursion (BFS/DFS + memo) over the identical edge set (dump calls_resolved/3 + inherits_resolved/2 + dep_raw/2 once from Prolog so both sides recurse over the same graph — resolver differences must not pollute the recursion comparison).
• Py-cli — end-to-end scripts/serialize_codebase_ast.py call-graph / find-cycles / who-imports timings (cold, its own load) — the "what an agent actually waits for" number.

Workloads (all recursive):

• W1 reverse-dependency closure (affected/2 ≡ transitive callers) for the 10 highest-fan-in symbols (pick by edge count) — depths deep enough to matter.
• W2 transitive module deps (mod_depends/2) both directions for 10 modules.
• W3 import-cycle enumeration (tool_cycles vs Python cycle finder vs find-cycles CLI) — canonicalized cycle sets compared.
• W4 dep_path between 10 distant module pairs.

Protocol: median of ≥5 timed runs per (mode, workload); 1 warmup discarded for warm modes; correctness parity asserted first (identical result sets for W1/W2/W4; canonical-set diff explained for W3) — a fast wrong answer loses. Output: scripts/benchmark_prolog_codeintel.py (rerunnable, argparse, no hardcoded paths) + reports/prolog-codeintel-benchmark.md with raw tables and per-workload winner+ratio.

Verdict rubric: the user's claim is assessed per deployment mode — "Prolog is better/faster than Python for recursive work" gets PASS/FAIL for P-sub and P-janus separately, with numbers. No hedging.

Waves (swarm)

• Wave 1 (parallel):
  • reviewer → adversarial review of the whole commit; confirm/extend D1–D10.
  • builder → benchmark harness + report (drives Prolog directly; does not depend on the buggy wrapper paths; may time the shipped wrapper as an extra row).
• Wave 2 (needs Wave-1 review): builder → fix D1–D9 per consolidated brief.
• Wave 3 (needs Wave 2): test-writer → tests/tools/test_codebase_tools.py + tests/scripts/test_benchmark_prolog_codeintel.py: real swipl, no mocks, function-scope fixtures, tmp_path fact/result files, small AST fixture generated at test time by serializing a tiny temp package.
• Wave 4 (me): gates (black/ruff/pyright/check-deps/test-review/code-review), affected tests solo, verdict synthesis, session logging, scoped commits on branch prolog-codeintel-verification.

Out of scope (backlogged, design decisions): registering CodebaseTools in the volk tool registry/MCP; replacing subprocess transport with janus in production.

Plan self-review — risks & mitigations

• Risk: janus consult of :- table rules misbehaves in 3.14t → benchmark harness treats P-janus as optional row with explicit failure note; claim then judged on P-sub only.
• Risk: parity mismatch Prolog vs Py-mem from set semantics (self-loops, dedup order) → compare as frozensets; investigate any delta before timing.
• Risk: Wave-2 fixes change wrapper API while Wave-1 benchmark times the shipped code → benchmark pins its own drivers; wrapper row re-timed after fixes if API changed.
• Risk: commit scope pollution (dirty working tree: .kimi/, docs regenerations) → scoped git add of named files only; review index before each commit.
• Risk: test runtime blowup (1.2 s/subprocess query) → tiny fixture AST for tests; full-size graph only in the benchmark script (not in the suite).

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks