Orchestrata Ultimate › Plans › prolog-utilization-plan

Prolog utilization plan — unused engine capabilities mapped to orchestrata

Source: plans/prolog-utilization-plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-10 · 262 lines.

Prose pending.

Plan text

Prolog utilization plan — unused engine capabilities mapped to orchestrata

Status: PROPOSED (2026-07-10). No implementation yet — this documents the options, their landing sites, effort/value, and a recommended sequence. Companion to the governance janus migration (session 165) and the SPEC-007 policy engine.

As-is: what Prolog does today

Prolog currently serves as a deny-by-default decision table:

• Governance: governance/policy_engine.py, constitutional.pl, ethical boundaries, resources.pl (may_access_file/network/api/db), escalation rules, safe-mode.
• Kernel: kernel/pam_enforcer.py, kernel/policy_evaluator.py (staged AUDIT-ONLY in the dispatcher).
• Orchestrator: intent authorization/validation (orchestrator/prolog_bridge.py, intent_router/validator Prolog paths).
• Session: session/prolog_facts.py, PrologSessionInterface (session state as facts).
• CXP: cxp/prolog_bridge.py (routing governance).
• Spec tooling: scripts/parse_spec_requirements.py (requirement checklists → ground facts).

Pattern: assert ground facts via janus, consult rules from .pl files, ask yes/no. Single-step inference only. The three Prolog containers (10.92.53.16/.17/.18) are essentially idle — janus does the real work in-process.

Standing constraints (do not violate)

1. Variable-bearing rules live in .pl files (consult); only GROUND facts are assertz'd — user ruling, session 165 (see memory: project-prolog-pyswip-load-bearing). Never reintroduce pyswip.
2. Deny-by-default everywhere a decision gate exists.
3. Test discipline applies: no mocks; real janus engine; the shared-engine isolation lessons from tests/governance/conftest.py (unload + abolish, user: shadowing) apply to every new rule file.
4. Specs are authoritative — each adopted item below should get a SPEC (next free numbers) before implementation.

The nine unutilized capabilities

1. Deductive reasoning over the knowledge graph (with tabling)

• What: mirror code-intelligence facts (calls/2, inherits/2, contains/2, imports/2) from the Neo4j KG into Prolog; write recursive rules for transitive impact, reachability, dead code, layering. SWI tabling (:- table affected/1.) makes transitive closure over a KG-sized call graph terminate with memoization.
• Replaces/augments: orchestrator/impact/{analyzer,dependency_analyzer, planner}.py (hand-written traversal), parts of scripts/architecture_checker.py (layer rules as facts+rules instead of Python constants).
• Design sketch: prolog/code_graph.pl (rules, tabled) + a fact loader fed by the existing AST sync (sync_ast_to_neo4j gains a --prolog-facts emitter, or facts are generated straight from plans/ast-ultimate.json). Facts are ground → assertz-safe; rules stay in the file.
• Value: high — replaces real Python complexity; enables "what breaks if I change X" as a query. Effort: M (fact emitter + rules + tests).
• Risk: fact-base staleness — regenerate alongside the AST snapshot (validate_ci gate 9 already forces AST freshness; extend it).

2. Constraint solving — CLP(FD)

• What: library(clpfd) for allocation/scheduling: test-batch bin packing (duration-aware), volk-pool assignment under capability + concurrency constraints, pipeline stage scheduling.
• Replaces/augments: the greedy LPT packer in core/threading/test_executor.py (_batch_modules), pool sizing in agents/pool.py.
• Value: medium — LPT is already good; CLP(FD) wins when constraints multiply (worker-specific Neo4j instances, per-module conflicts). Effort: M. Priority: P3 — adopt only when the greedy packer demonstrably misses.

3. Proof trees / explanations for governance decisions

• What: a small meta-interpreter (prove(Goal, ProofTree)) so every allow/deny carries its full derivation: which rule fired, which facts it used; plus "why-not" for denials (enumerate the failing preconditions of each candidate rule).
• Replaces/augments: today's AccessResult.reason strings ("prolog_policy" / "prolog_denied" — near-zero information; see the 2026-07-10 resource-access flake, which a proof tree would have made self-diagnosing).
• Landing sites: governance/prolog_client.py (optional explain=True query path), gateway audit writer (persist the proof tree JSON on the audit node), compliance volk (cite the derivation in reports).
• Design sketch: prolog/explain.pl meta-interpreter (~40 lines, classic vanilla-interpreter extension); janus marshals the tree as nested lists → JSON. No change to the fast non-explaining path.
• Value: high (audit + debugging + compliance). Effort: S/M.

4. DCGs for intent and spec parsing

• What: Definite Clause Grammars as a real grammar for user intents and requirement-checklist lines — structured, bidirectional (parse AND generate), testable clause by clause.
• Replaces/augments: regex/keyword extraction in orchestrator/intent_parser.py, orchestrator/nlp.py, orchestrator/parameter_extractor.py; the regex parser in scripts/parse_spec_requirements.py.
• Design sketch: prolog/intent_grammar.pl; Python keeps the current parser as fallback; DCG path behind a config flag until parity is proven against the existing parser test corpus.
• Value: medium-high (structure + maintainability). Effort: M/L (grammar coverage takes iteration).

5. Pipeline plan derivation and verification

• What: express pipeline stages as precondition/postcondition rules; derive legal stage orderings (STRIPS-style) instead of maintaining static WorkType→StageSpec tables; verify safety properties as queries ("compliance precedes merge on every path", "no retry cycle without an escalation exit", "every stage reachable").
• Replaces/augments: orchestrator/pipeline_router.py static tables (keep the tables as the source of truth initially — Prolog VERIFIES them; derivation can replace them later if it earns it).
• Value: medium-high (catches routing-table drift the moment it's introduced). Effort: S for verification-only; L for derivation.

6. Integrity constraints over the RTM / traceability

• What: traceability completeness as Prolog integrity constraints with counterexample enumeration: violation(req(R)) :- requirement(R), \+ (trace(R, T), test(T), proof_class(T, C), C >= min_class(R)). The engine returns the exact violating REQ IDs.
• Replaces/augments: the Python row-grading logic in scripts/generate_rtm_evidence.py and the check-traces ratchet (keep both; the Prolog layer adds declarative, queryable rules — e.g. "which SPEC-106 requirements are covered only by STRUCTURAL proof?").
• Facts already exist: parse_spec_requirements.py emits requirement facts today — this extends that pipeline with trace/test facts.
• Value: high, direct fit with the completed trace-remediation program. Effort: S/M.

7. Abductive diagnosis for test failures

• What: symptom facts (failure type, module, infra state, recent diffs)
  • causal rules → ranked root-cause hypotheses via abduction.
• Replaces/augments: the fixed 7-step procedure in scripts/diagnose_test_failure.py (keep the evidence-gathering steps — they BECOME the fact emitters; abduction replaces the hardcoded decision ladder).
• Value: medium. Effort: M. Good follow-on after #3's meta-interpreter exists (explanations of hypotheses come free).

8. Temporal / event-calculus reasoning over audit history

• What: event-calculus rules (holds_at/2, initiates/3, terminates/3) over the append-only session/gateway audit events, so the system can answer "which policies were in force when intent X was dispatched" and retroactive compliance queries.
• Landing sites: fact emitter from the Neo4j audit nodes; rules in prolog/audit_temporal.pl; surfaced through the compliance volk.
• Caveat: the audit data-quality debt (un-MERGEd/append-only Session nodes — see plans/context-tiers-design.md) must be priced in; garbage events → garbage inferences. Pair with the KG cleanup, not before it.
• Value: medium (audit/forensics). Effort: M/L.

9. Defeasible policy layering (defaults + exceptions)

• What: principled non-monotonic precedence — constitutional > ethical > resource > team norms — with explicit override semantics, replacing Python if-chains and ad-hoc override facts (fan_out_override/6).
• Landing sites: governance/policy_engine.py decision combination; prolog/policy_precedence.pl.
• Design sketch: standard defeasible pattern (allowed(A) :- default_allowed(A), \+ overridden(A). with layered overridden/1 rules); every layer's verdict recorded (pairs with #3).
• Value: medium-high (correctness of policy composition). Effort: M.

Infra note

The Prolog containers (.16/.17/.18) are the natural host for the long-lived, tabled fact bases (#1, #8) — an engine that survives process restarts and serves queries over MQI/socket, while janus stays the in-process path for per-decision governance (#3, #9). Do not build this split until #1 actually needs persistence; in-process janus with on-demand fact loading is the right first cut.

Extension: CLIPS — forward-chaining production system

CLIPS (C Language Integrated Production System, NASA-lineage expert-system shell, Rete algorithm; Python bindings via clipspy) complements rather than competes with Prolog. The division of labor is the classic one:

• Prolog (backward chaining) — query-driven: "is this allowed?", "what explains this?", "what does this depend on?". You ask; it derives. Stays THE engine for every item #1–#9 above.
• CLIPS (forward chaining, Rete) — data-driven: facts stream in, rules fire the instant their conditions match, with efficient incremental re-matching across thousands of rules. You feed it; it reacts.

Non-negotiable boundary: Prolog remains the single source of truth for governance/policy DECISIONS (SPEC-007 lineage). CLIPS never re-encodes policy rules — it may only consume derived facts (e.g. a deny event Prolog already issued) as inputs to reactive monitoring. Two engines answering the same policy question is a split-brain waiting to happen.

C1. Event-stream watchdog (pilot)

• What: subscribe a CLIPS session to the kernel/gateway event streams (kernel/event_bus.py, kernel/pipeline_events.py, gateway broadcast) as templated facts; Rete rules detect cross-event PATTERNS no single handler sees: retry storms ("≥3 retry events for one pipeline within N events"), stage stalls (start without matching finish while later stages progress), escalation loops, per-volk failure clusters.
• Why CLIPS: this is continuous incremental matching over a growing fact base — exactly what Rete is for; polling Prolog queries per event would re-derive from scratch each time.
• Landing: integrations/clips_watchdog.py (clipspy), rules in src/orchestrata_ultimate/clips/watchdog.clp; emits watchdog.* events back onto the bus (consumed by escalation / PM volk).
• Value: high (a class of defect nothing currently watches for). Effort: M.

C2. Resource / infra health correlation

• What: kernel/resource_monitor.py, provider health (providers/health_monitor.py, circuit breaker state), and start_test_services.py-style probes as facts; rules correlate ("provider X circuit open AND volk Y queue growing → pre-emptive reroute suggestion"). Today each monitor alerts in isolation.
• Value: medium-high. Effort: M. Builds on C1's fact plumbing.

C3. KG anomaly detection rules

• What: port/extend knowledge_graph/anomaly_detector.py (Spec 008 FR-010) heuristics into declarative CLIPS rules over graph-delta facts — new-node/edge streams from the sync engine fire anomaly rules incrementally instead of batch re-scans.
• Value: medium. Effort: M/L. Only after C1 proves the seam.

CLIPS integration mechanics

• Binding: clipspy under [project.optional-dependencies] (Key Rule: every dep declared; lazy import in the watchdog module).
• Host ground truth (verified 2026-07-10): clips 6.4.2-1 already installed via AUR at /usr/bin/clips (examples in /usr/share/clips/examples/); rule-firing smoke test green. clipspy 1.0.6 works on system CPython 3.14.6 (prebuilt wheel). BLOCKER for in-venv use: no clipspy wheel exists for free-threaded 3.14t (this project's interpreter) and the sdist build fails — it doesn't bundle CLIPS headers and no system libclips/headers package exists. C1 therefore has two viable shapes, decide at pilot time: (a) build libclips + headers from the 6.4.x core source so clipspy compiles under 3.14t (then in-venv as designed), or (b) run the watchdog as a SEPARATE system-python process subscribed over the event bus / Redis (matches its reactive, process-independent nature anyway; no project-venv dependency at all).
• Rules-as-files discipline: mirror the Prolog ruling — .clp files are the rule source (version-controlled, reviewed); Python only asserts ground facts and reads activations. Same engine-isolation care in tests (fresh Environment() per test; no shared module state).
• Tests: real clipspy Environment, real event fixtures; no doubles — identical standards to everything else in this suite.
• Sequencing: C1 piloted AFTER Prolog P1 lands (P1's fact-emission plumbing — KG facts, audit facts — is shared groundwork). If C1 doesn't pay for itself, C2/C3 die with it and Prolog #8 (event calculus) covers the retrospective slice of the same ground.

Recommended sequence

Each adopted item gets: a SPEC (next free number) with requirements + TST IDs before code; rules in .pl files under src/orchestrata_ultimate/ prolog/; ground-facts-only assertion; engine-isolation fixtures per the governance conftest pattern; ≥90% coverage like everything else.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks